Login
[x]
Log in using an account from:
Fedora Account System
Red Hat Associate
Red Hat Customer
Or login using a Red Hat Bugzilla account
Forgot Password
Login:
Hide Forgot
Create an Account
Red Hat Bugzilla – Attachment 941196 Details for
Bug 1146648
master domain down after upgrade, tracebacks in vdsm.log
[?]
New
Simple Search
Advanced Search
My Links
Browse
Requests
Reports
Current State
Search
Tabular reports
Graphical reports
Duplicates
Other Reports
User Changes
Plotly Reports
Bug Status
Bug Severity
Non-Defaults
|
Product Dashboard
Help
Page Help!
Bug Writing Guidelines
What's new
Browser Support Policy
5.0.4.rh83 Release notes
FAQ
Guides index
User guide
Web Services
Contact
Legal
This site requires JavaScript to be enabled to function correctly, please enable it.
vdsm log
vdsm.log (text/plain), 10.79 MB, created by
Petr Beňas
on 2014-09-25 16:41:15 UTC
(
hide
)
Description:
vdsm log
Filename:
MIME Type:
Creator:
Petr Beňas
Created:
2014-09-25 16:41:15 UTC
Size:
10.79 MB
patch
obsolete
>Thread-29::DEBUG::2014-09-25 17:01:01,179::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:01,179::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:01,180::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:01:01,181::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:01:01,182::__init__::232::IOProcess::(_processLogs) (106700) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:01,190::__init__::232::IOProcess::(_processLogs) (106700) Queuing response >Thread-80::DEBUG::2014-09-25 17:01:01,191::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000463712 s, 815 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:01,192::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:01,193::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:01,193::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:01,193::__init__::232::IOProcess::(_processLogs) (106701) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:01,194::__init__::232::IOProcess::(_processLogs) (106701) Queuing response >Dummy-148::DEBUG::2014-09-25 17:01:02,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:02,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193892 s, 52.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:03,113::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:03,114::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69377::DEBUG::2014-09-25 17:01:03,116::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:01:03,630::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:03,630::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:03,630::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:03,631::__init__::232::IOProcess::(_processLogs) (106702) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:03,631::__init__::232::IOProcess::(_processLogs) (106702) Queuing response >Thread-81::DEBUG::2014-09-25 17:01:03,632::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:01:03,639::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000281503 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:03,640::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:03,640::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:03,640::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:03,641::__init__::232::IOProcess::(_processLogs) (106703) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:03,642::__init__::232::IOProcess::(_processLogs) (106703) Queuing response >Dummy-148::DEBUG::2014-09-25 17:01:04,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:04,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0706617 s, 14.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:06,133::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:06,133::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69378::DEBUG::2014-09-25 17:01:06,135::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:06,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:06,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188115 s, 54.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:01:08,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:08,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199173 s, 51.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:08,300::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:08,300::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:08,300::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:08,300::__init__::232::IOProcess::(_processLogs) (106704) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:08,303::__init__::232::IOProcess::(_processLogs) (106704) Queuing response >Thread-79::DEBUG::2014-09-25 17:01:08,304::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:01:08,324::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00699089 s, 80.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:08,325::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:08,325::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:08,326::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:08,326::__init__::232::IOProcess::(_processLogs) (106705) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:08,331::__init__::232::IOProcess::(_processLogs) (106705) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:08,333::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:08,333::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:08,334::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:08,334::__init__::232::IOProcess::(_processLogs) (106706) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:08,335::__init__::232::IOProcess::(_processLogs) (106706) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:08,335::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:08,335::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:08,336::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:08,337::__init__::232::IOProcess::(_processLogs) (106707) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:08,337::__init__::232::IOProcess::(_processLogs) (106707) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:08,337::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:08,337::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:08,338::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:08,339::__init__::232::IOProcess::(_processLogs) (106708) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:08,339::__init__::232::IOProcess::(_processLogs) (106708) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:09,148::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:09,150::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69379::DEBUG::2014-09-25 17:01:09,153::task::595::Storage.TaskManager.Task::(_updateState) Task=`4675d0cb-c99e-4d2d-8564-3b36a097134b`::moving from state init -> state preparing >Thread-69379::INFO::2014-09-25 17:01:09,153::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69379::INFO::2014-09-25 17:01:09,153::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00699089', 'lastCheck': '0.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000463712', 'lastCheck': '8.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000281503', 'lastCheck': '5.5', 'valid': True}} >Thread-69379::DEBUG::2014-09-25 17:01:09,154::task::1191::Storage.TaskManager.Task::(prepare) Task=`4675d0cb-c99e-4d2d-8564-3b36a097134b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00699089', 'lastCheck': '0.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000463712', 'lastCheck': '8.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000281503', 'lastCheck': '5.5', 'valid': True}} >Thread-69379::DEBUG::2014-09-25 17:01:09,154::task::595::Storage.TaskManager.Task::(_updateState) Task=`4675d0cb-c99e-4d2d-8564-3b36a097134b`::moving from state preparing -> state finished >Thread-69379::DEBUG::2014-09-25 17:01:09,154::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69379::DEBUG::2014-09-25 17:01:09,154::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69379::DEBUG::2014-09-25 17:01:09,155::task::993::Storage.TaskManager.Task::(_decref) Task=`4675d0cb-c99e-4d2d-8564-3b36a097134b`::ref 0 aborting False >Thread-69379::DEBUG::2014-09-25 17:01:09,157::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:09,172::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:09,173::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69380::DEBUG::2014-09-25 17:01:09,175::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:10,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:10,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212967 s, 48.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:11,119::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69381::DEBUG::2014-09-25 17:01:11,120::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:01:11,120::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69381::DEBUG::2014-09-25 17:01:11,123::task::595::Storage.TaskManager.Task::(_updateState) Task=`43e3ba3a-4a10-4ee3-8129-5a02c3924e0d`::moving from state init -> state preparing >Thread-69381::INFO::2014-09-25 17:01:11,123::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69381::INFO::2014-09-25 17:01:11,124::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69381::DEBUG::2014-09-25 17:01:11,124::task::1191::Storage.TaskManager.Task::(prepare) Task=`43e3ba3a-4a10-4ee3-8129-5a02c3924e0d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69381::DEBUG::2014-09-25 17:01:11,124::task::595::Storage.TaskManager.Task::(_updateState) Task=`43e3ba3a-4a10-4ee3-8129-5a02c3924e0d`::moving from state preparing -> state finished >Thread-69381::DEBUG::2014-09-25 17:01:11,124::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69381::DEBUG::2014-09-25 17:01:11,125::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69381::DEBUG::2014-09-25 17:01:11,125::task::993::Storage.TaskManager.Task::(_decref) Task=`43e3ba3a-4a10-4ee3-8129-5a02c3924e0d`::ref 0 aborting False >Thread-69381::DEBUG::2014-09-25 17:01:11,125::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69381::DEBUG::2014-09-25 17:01:11,126::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:11,135::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69382::DEBUG::2014-09-25 17:01:11,136::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:01:11,137::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69382::DEBUG::2014-09-25 17:01:11,139::task::595::Storage.TaskManager.Task::(_updateState) Task=`b2b31ebe-f018-4a7b-b58f-afcb07c4f66a`::moving from state init -> state preparing >Thread-69382::INFO::2014-09-25 17:01:11,139::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69382::DEBUG::2014-09-25 17:01:11,140::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`82465aa0-2ce1-4d3d-bc59-dd9c4ffbcb0f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69382::DEBUG::2014-09-25 17:01:11,140::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69382::DEBUG::2014-09-25 17:01:11,140::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69382::DEBUG::2014-09-25 17:01:11,141::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`82465aa0-2ce1-4d3d-bc59-dd9c4ffbcb0f`::Granted request >Thread-69382::DEBUG::2014-09-25 17:01:11,141::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b2b31ebe-f018-4a7b-b58f-afcb07c4f66a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69382::DEBUG::2014-09-25 17:01:11,141::task::993::Storage.TaskManager.Task::(_decref) Task=`b2b31ebe-f018-4a7b-b58f-afcb07c4f66a`::ref 1 aborting False >Thread-69382::INFO::2014-09-25 17:01:11,146::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69382::DEBUG::2014-09-25 17:01:11,146::task::1191::Storage.TaskManager.Task::(prepare) Task=`b2b31ebe-f018-4a7b-b58f-afcb07c4f66a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69382::DEBUG::2014-09-25 17:01:11,146::task::595::Storage.TaskManager.Task::(_updateState) Task=`b2b31ebe-f018-4a7b-b58f-afcb07c4f66a`::moving from state preparing -> state finished >Thread-69382::DEBUG::2014-09-25 17:01:11,147::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69382::DEBUG::2014-09-25 17:01:11,147::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69382::DEBUG::2014-09-25 17:01:11,147::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69382::DEBUG::2014-09-25 17:01:11,147::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69382::DEBUG::2014-09-25 17:01:11,148::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69382::DEBUG::2014-09-25 17:01:11,148::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69382::DEBUG::2014-09-25 17:01:11,148::task::993::Storage.TaskManager.Task::(_decref) Task=`b2b31ebe-f018-4a7b-b58f-afcb07c4f66a`::ref 0 aborting False >Thread-69382::DEBUG::2014-09-25 17:01:11,149::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69382::DEBUG::2014-09-25 17:01:11,149::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:01:11,196::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:11,196::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:11,197::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:11,198::__init__::232::IOProcess::(_processLogs) (106709) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:01:11,198::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:01:11,199::__init__::232::IOProcess::(_processLogs) (106709) Queuing response >Thread-80::DEBUG::2014-09-25 17:01:11,208::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000553422 s, 683 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:11,209::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:11,209::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:11,210::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:11,210::__init__::232::IOProcess::(_processLogs) (106710) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:11,211::__init__::232::IOProcess::(_processLogs) (106710) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:12,216::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:12,218::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69383::DEBUG::2014-09-25 17:01:12,220::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:12,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:12,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145398 s, 70.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:13,642::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:13,643::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:13,643::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:13,643::__init__::232::IOProcess::(_processLogs) (106711) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:13,647::__init__::232::IOProcess::(_processLogs) (106711) Queuing response >Thread-81::DEBUG::2014-09-25 17:01:13,648::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:01:13,666::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00147581 s, 251 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:13,667::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:13,667::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:13,667::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:13,668::__init__::232::IOProcess::(_processLogs) (106712) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:13,673::__init__::232::IOProcess::(_processLogs) (106712) Queuing response >Dummy-148::DEBUG::2014-09-25 17:01:14,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:14,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192391 s, 53.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:15,235::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:15,237::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69384::DEBUG::2014-09-25 17:01:15,239::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:16,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:16,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0274252 s, 37.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:18,257::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:18,259::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69385::DEBUG::2014-09-25 17:01:18,261::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:01:18,339::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:18,340::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:18,340::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:18,340::__init__::232::IOProcess::(_processLogs) (106713) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:18,343::__init__::232::IOProcess::(_processLogs) (106713) Queuing response >Thread-79::DEBUG::2014-09-25 17:01:18,344::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:01:18,360::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00533196 s, 105 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:18,362::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:18,362::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:18,362::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:18,362::__init__::232::IOProcess::(_processLogs) (106714) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:18,367::__init__::232::IOProcess::(_processLogs) (106714) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:18,369::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:18,369::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:18,370::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:18,370::__init__::232::IOProcess::(_processLogs) (106715) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:18,370::__init__::232::IOProcess::(_processLogs) (106715) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:18,371::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:18,371::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:18,372::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:18,372::__init__::232::IOProcess::(_processLogs) (106716) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:18,372::__init__::232::IOProcess::(_processLogs) (106716) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:18,373::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:18,373::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:18,374::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:18,374::__init__::232::IOProcess::(_processLogs) (106717) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:18,375::__init__::232::IOProcess::(_processLogs) (106717) Queuing response >Dummy-148::DEBUG::2014-09-25 17:01:18,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:18,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021049 s, 48.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:01:20,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:20,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133959 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:21,175::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:21,176::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69386::DEBUG::2014-09-25 17:01:21,176::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69386::DEBUG::2014-09-25 17:01:21,178::task::595::Storage.TaskManager.Task::(_updateState) Task=`5aa3f92d-7726-4c25-933a-c21f6c1d5cb5`::moving from state init -> state preparing >Thread-69386::INFO::2014-09-25 17:01:21,178::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69386::INFO::2014-09-25 17:01:21,179::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69386::DEBUG::2014-09-25 17:01:21,179::task::1191::Storage.TaskManager.Task::(prepare) Task=`5aa3f92d-7726-4c25-933a-c21f6c1d5cb5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69386::DEBUG::2014-09-25 17:01:21,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`5aa3f92d-7726-4c25-933a-c21f6c1d5cb5`::moving from state preparing -> state finished >Thread-69386::DEBUG::2014-09-25 17:01:21,180::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69386::DEBUG::2014-09-25 17:01:21,180::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69386::DEBUG::2014-09-25 17:01:21,180::task::993::Storage.TaskManager.Task::(_decref) Task=`5aa3f92d-7726-4c25-933a-c21f6c1d5cb5`::ref 0 aborting False >Thread-69386::DEBUG::2014-09-25 17:01:21,180::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69386::DEBUG::2014-09-25 17:01:21,181::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:21,194::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:21,195::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69387::DEBUG::2014-09-25 17:01:21,195::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69387::DEBUG::2014-09-25 17:01:21,197::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c3da884-0f4e-4082-a019-0fe1a318b46c`::moving from state init -> state preparing >Thread-69387::INFO::2014-09-25 17:01:21,197::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69387::DEBUG::2014-09-25 17:01:21,198::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`092272e3-9a8c-4cc9-a906-92006e571aeb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69387::DEBUG::2014-09-25 17:01:21,198::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69387::DEBUG::2014-09-25 17:01:21,198::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69387::DEBUG::2014-09-25 17:01:21,199::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`092272e3-9a8c-4cc9-a906-92006e571aeb`::Granted request >Thread-69387::DEBUG::2014-09-25 17:01:21,199::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`6c3da884-0f4e-4082-a019-0fe1a318b46c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69387::DEBUG::2014-09-25 17:01:21,199::task::993::Storage.TaskManager.Task::(_decref) Task=`6c3da884-0f4e-4082-a019-0fe1a318b46c`::ref 1 aborting False >Thread-69387::INFO::2014-09-25 17:01:21,204::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69387::DEBUG::2014-09-25 17:01:21,204::task::1191::Storage.TaskManager.Task::(prepare) Task=`6c3da884-0f4e-4082-a019-0fe1a318b46c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69387::DEBUG::2014-09-25 17:01:21,204::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c3da884-0f4e-4082-a019-0fe1a318b46c`::moving from state preparing -> state finished >Thread-69387::DEBUG::2014-09-25 17:01:21,204::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69387::DEBUG::2014-09-25 17:01:21,205::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69387::DEBUG::2014-09-25 17:01:21,205::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69387::DEBUG::2014-09-25 17:01:21,205::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69387::DEBUG::2014-09-25 17:01:21,206::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69387::DEBUG::2014-09-25 17:01:21,206::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69387::DEBUG::2014-09-25 17:01:21,206::task::993::Storage.TaskManager.Task::(_decref) Task=`6c3da884-0f4e-4082-a019-0fe1a318b46c`::ref 0 aborting False >Thread-69387::DEBUG::2014-09-25 17:01:21,207::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69387::DEBUG::2014-09-25 17:01:21,207::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:01:21,213::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:21,213::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:21,213::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:21,213::__init__::232::IOProcess::(_processLogs) (106718) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:21,214::__init__::232::IOProcess::(_processLogs) (106718) Queuing response >Thread-80::DEBUG::2014-09-25 17:01:21,215::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:01:21,224::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000480361 s, 787 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:21,225::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:21,225::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:21,225::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:21,226::__init__::232::IOProcess::(_processLogs) (106719) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:21,226::__init__::232::IOProcess::(_processLogs) (106719) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:21,279::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:21,281::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69388::DEBUG::2014-09-25 17:01:21,283::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:22,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:22,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0278348 s, 36.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:23,675::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:23,676::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:23,676::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:23,676::__init__::232::IOProcess::(_processLogs) (106720) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:23,679::__init__::232::IOProcess::(_processLogs) (106720) Queuing response >Thread-81::DEBUG::2014-09-25 17:01:23,680::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:01:23,695::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00364394 s, 102 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:23,696::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:23,696::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:23,696::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:23,697::__init__::232::IOProcess::(_processLogs) (106721) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:23,697::__init__::232::IOProcess::(_processLogs) (106721) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:24,294::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:24,296::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69389::DEBUG::2014-09-25 17:01:24,299::task::595::Storage.TaskManager.Task::(_updateState) Task=`303d86da-bb88-488a-b846-f76565ee210a`::moving from state init -> state preparing >Thread-69389::INFO::2014-09-25 17:01:24,299::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69389::INFO::2014-09-25 17:01:24,299::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00533196', 'lastCheck': '5.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000480361', 'lastCheck': '3.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00364394', 'lastCheck': '0.6', 'valid': True}} >Thread-69389::DEBUG::2014-09-25 17:01:24,300::task::1191::Storage.TaskManager.Task::(prepare) Task=`303d86da-bb88-488a-b846-f76565ee210a`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00533196', 'lastCheck': '5.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000480361', 'lastCheck': '3.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00364394', 'lastCheck': '0.6', 'valid': True}} >Thread-69389::DEBUG::2014-09-25 17:01:24,300::task::595::Storage.TaskManager.Task::(_updateState) Task=`303d86da-bb88-488a-b846-f76565ee210a`::moving from state preparing -> state finished >Thread-69389::DEBUG::2014-09-25 17:01:24,300::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69389::DEBUG::2014-09-25 17:01:24,300::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69389::DEBUG::2014-09-25 17:01:24,301::task::993::Storage.TaskManager.Task::(_decref) Task=`303d86da-bb88-488a-b846-f76565ee210a`::ref 0 aborting False >Thread-69389::DEBUG::2014-09-25 17:01:24,303::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:24,321::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:24,322::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69390::DEBUG::2014-09-25 17:01:24,324::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:24,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:24,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164966 s, 62.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:01:26,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:26,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143278 s, 71.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:27,375::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:27,377::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69391::DEBUG::2014-09-25 17:01:27,383::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:01:28,375::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:28,375::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:28,376::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:28,376::__init__::232::IOProcess::(_processLogs) (106722) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:28,379::__init__::232::IOProcess::(_processLogs) (106722) Queuing response >Thread-79::DEBUG::2014-09-25 17:01:28,379::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:01:28,399::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00778581 s, 72.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:28,400::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:28,400::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:28,400::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:28,400::__init__::232::IOProcess::(_processLogs) (106723) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:28,407::__init__::232::IOProcess::(_processLogs) (106723) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:28,409::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:28,409::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:28,410::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:28,410::__init__::232::IOProcess::(_processLogs) (106724) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:28,411::__init__::232::IOProcess::(_processLogs) (106724) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:28,411::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:28,412::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:28,412::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:28,413::__init__::232::IOProcess::(_processLogs) (106725) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:28,413::__init__::232::IOProcess::(_processLogs) (106725) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:28,413::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:28,414::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:28,414::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:28,415::__init__::232::IOProcess::(_processLogs) (106726) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:28,415::__init__::232::IOProcess::(_processLogs) (106726) Queuing response >Dummy-148::DEBUG::2014-09-25 17:01:28,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:29,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.229329 s, 4.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:30,400::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:30,402::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69392::DEBUG::2014-09-25 17:01:30,404::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:31,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:31,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210098 s, 48.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:31,228::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:31,229::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:31,229::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:31,229::__init__::232::IOProcess::(_processLogs) (106727) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:31,230::__init__::232::IOProcess::(_processLogs) (106727) Queuing response >Thread-80::DEBUG::2014-09-25 17:01:31,231::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:31,239::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-80::DEBUG::2014-09-25 17:01:31,241::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000350096 s, 1.1 MB/s\n'; <rc> = 0 >JsonRpcServer::DEBUG::2014-09-25 17:01:31,242::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69393::DEBUG::2014-09-25 17:01:31,242::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-29::DEBUG::2014-09-25 17:01:31,245::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:31,246::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:31,247::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:31,248::__init__::232::IOProcess::(_processLogs) (106728) Got request for method 'statvfs' >Thread-69393::DEBUG::2014-09-25 17:01:31,249::task::595::Storage.TaskManager.Task::(_updateState) Task=`5c647413-67ff-4538-9092-e31f13f60119`::moving from state init -> state preparing >Thread-29::DEBUG::2014-09-25 17:01:31,251::__init__::232::IOProcess::(_processLogs) (106728) Queuing response >Thread-69393::INFO::2014-09-25 17:01:31,252::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69393::INFO::2014-09-25 17:01:31,253::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69393::DEBUG::2014-09-25 17:01:31,254::task::1191::Storage.TaskManager.Task::(prepare) Task=`5c647413-67ff-4538-9092-e31f13f60119`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69393::DEBUG::2014-09-25 17:01:31,254::task::595::Storage.TaskManager.Task::(_updateState) Task=`5c647413-67ff-4538-9092-e31f13f60119`::moving from state preparing -> state finished >Thread-69393::DEBUG::2014-09-25 17:01:31,254::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69393::DEBUG::2014-09-25 17:01:31,254::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69393::DEBUG::2014-09-25 17:01:31,254::task::993::Storage.TaskManager.Task::(_decref) Task=`5c647413-67ff-4538-9092-e31f13f60119`::ref 0 aborting False >Thread-69393::DEBUG::2014-09-25 17:01:31,255::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69393::DEBUG::2014-09-25 17:01:31,255::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:31,265::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:31,267::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69394::DEBUG::2014-09-25 17:01:31,267::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69394::DEBUG::2014-09-25 17:01:31,268::task::595::Storage.TaskManager.Task::(_updateState) Task=`7645bd12-ce7c-4085-8fe9-71f55bb386ad`::moving from state init -> state preparing >Thread-69394::INFO::2014-09-25 17:01:31,269::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69394::DEBUG::2014-09-25 17:01:31,269::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0f0eaf20-d63e-41d7-9767-2e79fde2168e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69394::DEBUG::2014-09-25 17:01:31,270::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69394::DEBUG::2014-09-25 17:01:31,270::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69394::DEBUG::2014-09-25 17:01:31,270::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0f0eaf20-d63e-41d7-9767-2e79fde2168e`::Granted request >Thread-69394::DEBUG::2014-09-25 17:01:31,271::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7645bd12-ce7c-4085-8fe9-71f55bb386ad`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69394::DEBUG::2014-09-25 17:01:31,271::task::993::Storage.TaskManager.Task::(_decref) Task=`7645bd12-ce7c-4085-8fe9-71f55bb386ad`::ref 1 aborting False >Thread-69394::INFO::2014-09-25 17:01:31,275::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69394::DEBUG::2014-09-25 17:01:31,276::task::1191::Storage.TaskManager.Task::(prepare) Task=`7645bd12-ce7c-4085-8fe9-71f55bb386ad`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69394::DEBUG::2014-09-25 17:01:31,276::task::595::Storage.TaskManager.Task::(_updateState) Task=`7645bd12-ce7c-4085-8fe9-71f55bb386ad`::moving from state preparing -> state finished >Thread-69394::DEBUG::2014-09-25 17:01:31,276::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69394::DEBUG::2014-09-25 17:01:31,276::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69394::DEBUG::2014-09-25 17:01:31,277::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69394::DEBUG::2014-09-25 17:01:31,277::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69394::DEBUG::2014-09-25 17:01:31,277::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69394::DEBUG::2014-09-25 17:01:31,278::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69394::DEBUG::2014-09-25 17:01:31,278::task::993::Storage.TaskManager.Task::(_decref) Task=`7645bd12-ce7c-4085-8fe9-71f55bb386ad`::ref 0 aborting False >Thread-69394::DEBUG::2014-09-25 17:01:31,278::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69394::DEBUG::2014-09-25 17:01:31,279::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:33,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:33,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145179 s, 70.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:33,417::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:33,419::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69395::DEBUG::2014-09-25 17:01:33,420::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:01:33,699::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:33,699::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:33,700::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:33,700::__init__::232::IOProcess::(_processLogs) (106729) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:33,702::__init__::232::IOProcess::(_processLogs) (106729) Queuing response >Thread-81::DEBUG::2014-09-25 17:01:33,703::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:01:33,714::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00164562 s, 225 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:33,715::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:33,716::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:33,716::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:33,716::__init__::232::IOProcess::(_processLogs) (106730) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:33,719::__init__::232::IOProcess::(_processLogs) (106730) Queuing response >Dummy-148::DEBUG::2014-09-25 17:01:35,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:35,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180061 s, 56.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:36,432::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:36,434::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69396::DEBUG::2014-09-25 17:01:36,436::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:37,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:37,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020188 s, 50.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:38,416::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:38,416::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:38,417::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:38,417::__init__::232::IOProcess::(_processLogs) (106731) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:38,452::__init__::232::IOProcess::(_processLogs) (106731) Queuing response >Thread-79::DEBUG::2014-09-25 17:01:38,453::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:01:38,481::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0086248 s, 65.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:38,482::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:38,482::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:38,482::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:38,483::__init__::232::IOProcess::(_processLogs) (106732) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:38,490::__init__::232::IOProcess::(_processLogs) (106732) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:38,491::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:38,492::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:38,492::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:38,492::__init__::232::IOProcess::(_processLogs) (106733) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:38,501::__init__::232::IOProcess::(_processLogs) (106733) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:38,502::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:38,502::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:38,502::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:38,503::__init__::232::IOProcess::(_processLogs) (106734) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:38,509::__init__::232::IOProcess::(_processLogs) (106734) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:38,510::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:38,510::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:38,511::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:38,511::__init__::232::IOProcess::(_processLogs) (106735) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:38,517::__init__::232::IOProcess::(_processLogs) (106735) Queuing response >Dummy-148::DEBUG::2014-09-25 17:01:39,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:39,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214336 s, 47.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:39,449::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:39,451::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69397::DEBUG::2014-09-25 17:01:39,453::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ed6debe-8a74-44de-9817-2714e16eca54`::moving from state init -> state preparing >Thread-69397::INFO::2014-09-25 17:01:39,454::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69397::INFO::2014-09-25 17:01:39,454::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0086248', 'lastCheck': '0.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000350096', 'lastCheck': '8.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00164562', 'lastCheck': '5.7', 'valid': True}} >Thread-69397::DEBUG::2014-09-25 17:01:39,454::task::1191::Storage.TaskManager.Task::(prepare) Task=`0ed6debe-8a74-44de-9817-2714e16eca54`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0086248', 'lastCheck': '0.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000350096', 'lastCheck': '8.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00164562', 'lastCheck': '5.7', 'valid': True}} >Thread-69397::DEBUG::2014-09-25 17:01:39,455::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ed6debe-8a74-44de-9817-2714e16eca54`::moving from state preparing -> state finished >Thread-69397::DEBUG::2014-09-25 17:01:39,455::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69397::DEBUG::2014-09-25 17:01:39,455::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69397::DEBUG::2014-09-25 17:01:39,455::task::993::Storage.TaskManager.Task::(_decref) Task=`0ed6debe-8a74-44de-9817-2714e16eca54`::ref 0 aborting False >Thread-69397::DEBUG::2014-09-25 17:01:39,458::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:39,481::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:39,482::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69398::DEBUG::2014-09-25 17:01:39,484::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:01:41,254::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:41,254::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:41,254::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:41,254::__init__::232::IOProcess::(_processLogs) (106736) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:41,255::__init__::232::IOProcess::(_processLogs) (106736) Queuing response >Thread-80::DEBUG::2014-09-25 17:01:41,257::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:01:41,266::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000538485 s, 702 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:41,267::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:41,268::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:41,268::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:41,268::__init__::232::IOProcess::(_processLogs) (106737) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:41,269::__init__::232::IOProcess::(_processLogs) (106737) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:41,306::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69399::DEBUG::2014-09-25 17:01:41,308::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:01:41,307::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69399::DEBUG::2014-09-25 17:01:41,310::task::595::Storage.TaskManager.Task::(_updateState) Task=`9df36a77-233d-4f4e-8914-cea321ec8d9e`::moving from state init -> state preparing >Thread-69399::INFO::2014-09-25 17:01:41,310::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69399::INFO::2014-09-25 17:01:41,311::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69399::DEBUG::2014-09-25 17:01:41,311::task::1191::Storage.TaskManager.Task::(prepare) Task=`9df36a77-233d-4f4e-8914-cea321ec8d9e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69399::DEBUG::2014-09-25 17:01:41,311::task::595::Storage.TaskManager.Task::(_updateState) Task=`9df36a77-233d-4f4e-8914-cea321ec8d9e`::moving from state preparing -> state finished >Thread-69399::DEBUG::2014-09-25 17:01:41,311::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69399::DEBUG::2014-09-25 17:01:41,312::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69399::DEBUG::2014-09-25 17:01:41,312::task::993::Storage.TaskManager.Task::(_decref) Task=`9df36a77-233d-4f4e-8914-cea321ec8d9e`::ref 0 aborting False >Thread-69399::DEBUG::2014-09-25 17:01:41,312::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69399::DEBUG::2014-09-25 17:01:41,313::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:41,327::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69400::DEBUG::2014-09-25 17:01:41,328::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:01:41,328::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69400::DEBUG::2014-09-25 17:01:41,330::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f1d84b5-6800-4acd-b552-5ba91e3d3976`::moving from state init -> state preparing >Thread-69400::INFO::2014-09-25 17:01:41,330::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69400::DEBUG::2014-09-25 17:01:41,332::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ee626cc5-c39e-4132-a4eb-a01e7922198e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69400::DEBUG::2014-09-25 17:01:41,332::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69400::DEBUG::2014-09-25 17:01:41,333::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69400::DEBUG::2014-09-25 17:01:41,333::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ee626cc5-c39e-4132-a4eb-a01e7922198e`::Granted request >Thread-69400::DEBUG::2014-09-25 17:01:41,333::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0f1d84b5-6800-4acd-b552-5ba91e3d3976`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69400::DEBUG::2014-09-25 17:01:41,334::task::993::Storage.TaskManager.Task::(_decref) Task=`0f1d84b5-6800-4acd-b552-5ba91e3d3976`::ref 1 aborting False >Thread-69400::INFO::2014-09-25 17:01:41,338::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69400::DEBUG::2014-09-25 17:01:41,338::task::1191::Storage.TaskManager.Task::(prepare) Task=`0f1d84b5-6800-4acd-b552-5ba91e3d3976`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69400::DEBUG::2014-09-25 17:01:41,339::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f1d84b5-6800-4acd-b552-5ba91e3d3976`::moving from state preparing -> state finished >Thread-69400::DEBUG::2014-09-25 17:01:41,339::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69400::DEBUG::2014-09-25 17:01:41,339::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69400::DEBUG::2014-09-25 17:01:41,339::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69400::DEBUG::2014-09-25 17:01:41,340::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69400::DEBUG::2014-09-25 17:01:41,340::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69400::DEBUG::2014-09-25 17:01:41,340::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69400::DEBUG::2014-09-25 17:01:41,341::task::993::Storage.TaskManager.Task::(_decref) Task=`0f1d84b5-6800-4acd-b552-5ba91e3d3976`::ref 0 aborting False >Thread-69400::DEBUG::2014-09-25 17:01:41,341::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69400::DEBUG::2014-09-25 17:01:41,341::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:41,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:41,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129306 s, 79.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:42,547::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:42,548::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69401::DEBUG::2014-09-25 17:01:42,551::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:43,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:43,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157577 s, 65.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:43,721::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:43,721::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:43,721::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:43,722::__init__::232::IOProcess::(_processLogs) (106738) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:43,724::__init__::232::IOProcess::(_processLogs) (106738) Queuing response >Thread-81::DEBUG::2014-09-25 17:01:43,725::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:01:43,735::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000692625 s, 536 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:43,736::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:43,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:43,737::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:43,737::__init__::232::IOProcess::(_processLogs) (106739) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:43,739::__init__::232::IOProcess::(_processLogs) (106739) Queuing response >Dummy-148::DEBUG::2014-09-25 17:01:45,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:45,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130185 s, 78.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:45,566::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:45,568::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69402::DEBUG::2014-09-25 17:01:45,570::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:47,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:47,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154783 s, 66.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:48,519::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:48,519::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:48,519::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:48,520::__init__::232::IOProcess::(_processLogs) (106740) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:48,520::__init__::232::IOProcess::(_processLogs) (106740) Queuing response >Thread-79::DEBUG::2014-09-25 17:01:48,521::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:01:48,530::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000455685 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:48,532::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:48,532::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:48,532::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:48,532::__init__::232::IOProcess::(_processLogs) (106741) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:48,533::__init__::232::IOProcess::(_processLogs) (106741) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:48,534::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:48,535::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:48,535::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:48,536::__init__::232::IOProcess::(_processLogs) (106742) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:48,536::__init__::232::IOProcess::(_processLogs) (106742) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:48,536::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:48,537::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:48,537::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:48,538::__init__::232::IOProcess::(_processLogs) (106743) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:48,538::__init__::232::IOProcess::(_processLogs) (106743) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:48,539::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:48,539::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:48,540::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:48,540::__init__::232::IOProcess::(_processLogs) (106744) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:48,540::__init__::232::IOProcess::(_processLogs) (106744) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:48,586::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:48,587::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69403::DEBUG::2014-09-25 17:01:48,589::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:49,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:49,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0216605 s, 47.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:51,271::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:51,272::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:51,272::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:51,272::__init__::232::IOProcess::(_processLogs) (106745) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:51,273::__init__::232::IOProcess::(_processLogs) (106745) Queuing response >Thread-80::DEBUG::2014-09-25 17:01:51,274::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:01:51,283::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000580575 s, 651 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:51,284::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:51,284::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:51,285::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:51,285::__init__::232::IOProcess::(_processLogs) (106746) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:51,285::__init__::232::IOProcess::(_processLogs) (106746) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:51,368::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:51,370::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69404::DEBUG::2014-09-25 17:01:51,370::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69404::DEBUG::2014-09-25 17:01:51,372::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5c887bb-a04b-480b-a32e-03b5430899ec`::moving from state init -> state preparing >Thread-69404::INFO::2014-09-25 17:01:51,372::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69404::INFO::2014-09-25 17:01:51,373::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69404::DEBUG::2014-09-25 17:01:51,373::task::1191::Storage.TaskManager.Task::(prepare) Task=`a5c887bb-a04b-480b-a32e-03b5430899ec`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69404::DEBUG::2014-09-25 17:01:51,373::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5c887bb-a04b-480b-a32e-03b5430899ec`::moving from state preparing -> state finished >Thread-69404::DEBUG::2014-09-25 17:01:51,374::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69404::DEBUG::2014-09-25 17:01:51,374::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69404::DEBUG::2014-09-25 17:01:51,374::task::993::Storage.TaskManager.Task::(_decref) Task=`a5c887bb-a04b-480b-a32e-03b5430899ec`::ref 0 aborting False >Thread-69404::DEBUG::2014-09-25 17:01:51,374::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69404::DEBUG::2014-09-25 17:01:51,375::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:51,389::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:51,390::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69405::DEBUG::2014-09-25 17:01:51,390::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69405::DEBUG::2014-09-25 17:01:51,392::task::595::Storage.TaskManager.Task::(_updateState) Task=`10fdeab3-3189-488d-9b4c-0d9756b0f47f`::moving from state init -> state preparing >Thread-69405::INFO::2014-09-25 17:01:51,392::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69405::DEBUG::2014-09-25 17:01:51,393::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e4c01501-bfc8-48fc-97e9-9dac3d61c5c7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69405::DEBUG::2014-09-25 17:01:51,393::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69405::DEBUG::2014-09-25 17:01:51,393::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69405::DEBUG::2014-09-25 17:01:51,394::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e4c01501-bfc8-48fc-97e9-9dac3d61c5c7`::Granted request >Thread-69405::DEBUG::2014-09-25 17:01:51,394::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`10fdeab3-3189-488d-9b4c-0d9756b0f47f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69405::DEBUG::2014-09-25 17:01:51,394::task::993::Storage.TaskManager.Task::(_decref) Task=`10fdeab3-3189-488d-9b4c-0d9756b0f47f`::ref 1 aborting False >Thread-69405::INFO::2014-09-25 17:01:51,399::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69405::DEBUG::2014-09-25 17:01:51,399::task::1191::Storage.TaskManager.Task::(prepare) Task=`10fdeab3-3189-488d-9b4c-0d9756b0f47f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69405::DEBUG::2014-09-25 17:01:51,399::task::595::Storage.TaskManager.Task::(_updateState) Task=`10fdeab3-3189-488d-9b4c-0d9756b0f47f`::moving from state preparing -> state finished >Thread-69405::DEBUG::2014-09-25 17:01:51,399::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69405::DEBUG::2014-09-25 17:01:51,400::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69405::DEBUG::2014-09-25 17:01:51,400::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69405::DEBUG::2014-09-25 17:01:51,400::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69405::DEBUG::2014-09-25 17:01:51,401::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69405::DEBUG::2014-09-25 17:01:51,401::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69405::DEBUG::2014-09-25 17:01:51,401::task::993::Storage.TaskManager.Task::(_decref) Task=`10fdeab3-3189-488d-9b4c-0d9756b0f47f`::ref 0 aborting False >Thread-69405::DEBUG::2014-09-25 17:01:51,402::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69405::DEBUG::2014-09-25 17:01:51,402::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:51,607::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:51,608::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69406::DEBUG::2014-09-25 17:01:51,610::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:51,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:51,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0219551 s, 46.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:01:53,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:01:53,742::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:53,744::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:53,745::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:53,746::__init__::232::IOProcess::(_processLogs) (106747) Got request for method 'statvfs' >Dummy-148::DEBUG::2014-09-25 17:01:53,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0238303 s, 43.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:53,754::__init__::232::IOProcess::(_processLogs) (106747) Queuing response >Thread-81::DEBUG::2014-09-25 17:01:53,756::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:01:53,781::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00278863 s, 133 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:53,782::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:53,782::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:53,783::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:53,783::__init__::232::IOProcess::(_processLogs) (106748) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:53,789::__init__::232::IOProcess::(_processLogs) (106748) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:54,625::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:54,627::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69407::DEBUG::2014-09-25 17:01:54,630::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f384294-8b57-4434-876e-633b8968942b`::moving from state init -> state preparing >Thread-69407::INFO::2014-09-25 17:01:54,630::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69407::INFO::2014-09-25 17:01:54,630::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000455685', 'lastCheck': '6.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000580575', 'lastCheck': '3.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00278863', 'lastCheck': '0.8', 'valid': True}} >Thread-69407::DEBUG::2014-09-25 17:01:54,631::task::1191::Storage.TaskManager.Task::(prepare) Task=`4f384294-8b57-4434-876e-633b8968942b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000455685', 'lastCheck': '6.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000580575', 'lastCheck': '3.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00278863', 'lastCheck': '0.8', 'valid': True}} >Thread-69407::DEBUG::2014-09-25 17:01:54,631::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f384294-8b57-4434-876e-633b8968942b`::moving from state preparing -> state finished >Thread-69407::DEBUG::2014-09-25 17:01:54,631::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69407::DEBUG::2014-09-25 17:01:54,631::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69407::DEBUG::2014-09-25 17:01:54,632::task::993::Storage.TaskManager.Task::(_decref) Task=`4f384294-8b57-4434-876e-633b8968942b`::ref 0 aborting False >Thread-69407::DEBUG::2014-09-25 17:01:54,634::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:54,655::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:54,656::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69408::DEBUG::2014-09-25 17:01:54,658::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:55,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:55,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131583 s, 77.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:01:57,715::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:01:57,717::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69409::DEBUG::2014-09-25 17:01:57,719::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:01:57,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:57,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151028 s, 67.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:58,541::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:58,541::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:58,541::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:58,542::__init__::232::IOProcess::(_processLogs) (106749) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:58,544::__init__::232::IOProcess::(_processLogs) (106749) Queuing response >Thread-79::DEBUG::2014-09-25 17:01:58,545::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:01:58,562::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00602192 s, 93.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:01:58,563::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:58,563::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:58,563::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:58,564::__init__::232::IOProcess::(_processLogs) (106750) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:01:58,565::__init__::232::IOProcess::(_processLogs) (106750) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:58,566::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:58,566::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:58,567::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:58,567::__init__::232::IOProcess::(_processLogs) (106751) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:58,568::__init__::232::IOProcess::(_processLogs) (106751) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:58,568::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:58,568::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:58,569::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:58,569::__init__::232::IOProcess::(_processLogs) (106752) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:58,570::__init__::232::IOProcess::(_processLogs) (106752) Queuing response >Thread-29::DEBUG::2014-09-25 17:01:58,570::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:01:58,571::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:01:58,571::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:01:58,572::__init__::232::IOProcess::(_processLogs) (106753) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:01:58,572::__init__::232::IOProcess::(_processLogs) (106753) Queuing response >Dummy-148::DEBUG::2014-09-25 17:01:59,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:01:59,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169149 s, 60.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:00,735::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:00,737::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69410::DEBUG::2014-09-25 17:02:00,739::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:02:01,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:01,288::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:01,288::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:01,289::__init__::232::IOProcess::(_processLogs) (106754) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:01,289::__init__::232::IOProcess::(_processLogs) (106754) Queuing response >Thread-80::DEBUG::2014-09-25 17:02:01,290::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:02:01,299::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.0004888 s, 773 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:01,301::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:01,301::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:01,301::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:01,301::__init__::232::IOProcess::(_processLogs) (106755) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:01,302::__init__::232::IOProcess::(_processLogs) (106755) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:01,434::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69411::DEBUG::2014-09-25 17:02:01,435::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:02:01,435::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69411::DEBUG::2014-09-25 17:02:01,438::task::595::Storage.TaskManager.Task::(_updateState) Task=`7c254ea9-43fc-464e-b3cd-418334cdaa57`::moving from state init -> state preparing >Thread-69411::INFO::2014-09-25 17:02:01,438::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69411::INFO::2014-09-25 17:02:01,438::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69411::DEBUG::2014-09-25 17:02:01,439::task::1191::Storage.TaskManager.Task::(prepare) Task=`7c254ea9-43fc-464e-b3cd-418334cdaa57`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69411::DEBUG::2014-09-25 17:02:01,439::task::595::Storage.TaskManager.Task::(_updateState) Task=`7c254ea9-43fc-464e-b3cd-418334cdaa57`::moving from state preparing -> state finished >Thread-69411::DEBUG::2014-09-25 17:02:01,439::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69411::DEBUG::2014-09-25 17:02:01,439::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69411::DEBUG::2014-09-25 17:02:01,440::task::993::Storage.TaskManager.Task::(_decref) Task=`7c254ea9-43fc-464e-b3cd-418334cdaa57`::ref 0 aborting False >Thread-69411::DEBUG::2014-09-25 17:02:01,440::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69411::DEBUG::2014-09-25 17:02:01,440::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:01,453::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69412::DEBUG::2014-09-25 17:02:01,455::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:02:01,455::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69412::DEBUG::2014-09-25 17:02:01,457::task::595::Storage.TaskManager.Task::(_updateState) Task=`1d17d3d2-d743-4b92-a8ed-6e3685dae572`::moving from state init -> state preparing >Thread-69412::INFO::2014-09-25 17:02:01,457::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69412::DEBUG::2014-09-25 17:02:01,458::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c5302206-19ca-4fe1-923f-7bfcff611bdb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69412::DEBUG::2014-09-25 17:02:01,458::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69412::DEBUG::2014-09-25 17:02:01,459::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69412::DEBUG::2014-09-25 17:02:01,459::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c5302206-19ca-4fe1-923f-7bfcff611bdb`::Granted request >Thread-69412::DEBUG::2014-09-25 17:02:01,459::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1d17d3d2-d743-4b92-a8ed-6e3685dae572`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69412::DEBUG::2014-09-25 17:02:01,460::task::993::Storage.TaskManager.Task::(_decref) Task=`1d17d3d2-d743-4b92-a8ed-6e3685dae572`::ref 1 aborting False >Thread-69412::INFO::2014-09-25 17:02:01,464::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69412::DEBUG::2014-09-25 17:02:01,464::task::1191::Storage.TaskManager.Task::(prepare) Task=`1d17d3d2-d743-4b92-a8ed-6e3685dae572`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69412::DEBUG::2014-09-25 17:02:01,464::task::595::Storage.TaskManager.Task::(_updateState) Task=`1d17d3d2-d743-4b92-a8ed-6e3685dae572`::moving from state preparing -> state finished >Thread-69412::DEBUG::2014-09-25 17:02:01,465::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69412::DEBUG::2014-09-25 17:02:01,465::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69412::DEBUG::2014-09-25 17:02:01,465::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69412::DEBUG::2014-09-25 17:02:01,466::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69412::DEBUG::2014-09-25 17:02:01,466::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69412::DEBUG::2014-09-25 17:02:01,466::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69412::DEBUG::2014-09-25 17:02:01,466::task::993::Storage.TaskManager.Task::(_decref) Task=`1d17d3d2-d743-4b92-a8ed-6e3685dae572`::ref 0 aborting False >Thread-69412::DEBUG::2014-09-25 17:02:01,467::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69412::DEBUG::2014-09-25 17:02:01,467::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:01,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:01,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145203 s, 70.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:03,754::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:03,756::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69413::DEBUG::2014-09-25 17:02:03,758::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:02:03,791::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:03,791::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:03,792::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:03,792::__init__::232::IOProcess::(_processLogs) (106756) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:03,797::__init__::232::IOProcess::(_processLogs) (106756) Queuing response >Thread-81::DEBUG::2014-09-25 17:02:03,798::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:02:03,816::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00552526 s, 67.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:03,817::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:03,817::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:03,818::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:03,818::__init__::232::IOProcess::(_processLogs) (106757) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:03,824::__init__::232::IOProcess::(_processLogs) (106757) Queuing response >Dummy-148::DEBUG::2014-09-25 17:02:03,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:03,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015 s, 68.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:02:06,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:06,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133911 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:06,774::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:06,776::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69414::DEBUG::2014-09-25 17:02:06,778::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:08,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:08,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162847 s, 62.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:08,573::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:08,573::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:08,573::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:08,574::__init__::232::IOProcess::(_processLogs) (106758) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:08,582::__init__::232::IOProcess::(_processLogs) (106758) Queuing response >Thread-79::DEBUG::2014-09-25 17:02:08,583::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:02:08,599::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00188026 s, 299 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:08,600::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:08,601::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:08,601::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:08,601::__init__::232::IOProcess::(_processLogs) (106759) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:08,606::__init__::232::IOProcess::(_processLogs) (106759) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:08,607::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:08,608::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:08,608::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:08,609::__init__::232::IOProcess::(_processLogs) (106760) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:08,609::__init__::232::IOProcess::(_processLogs) (106760) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:08,609::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:08,609::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:08,610::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:08,611::__init__::232::IOProcess::(_processLogs) (106761) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:08,611::__init__::232::IOProcess::(_processLogs) (106761) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:08,611::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:08,612::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:08,613::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:08,613::__init__::232::IOProcess::(_processLogs) (106762) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:08,613::__init__::232::IOProcess::(_processLogs) (106762) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:09,793::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:09,795::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69415::DEBUG::2014-09-25 17:02:09,797::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d982c59-02a7-421c-bd66-13173c216461`::moving from state init -> state preparing >Thread-69415::INFO::2014-09-25 17:02:09,797::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69415::INFO::2014-09-25 17:02:09,798::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00188026', 'lastCheck': '1.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0004888', 'lastCheck': '8.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00552526', 'lastCheck': '6.0', 'valid': True}} >Thread-69415::DEBUG::2014-09-25 17:02:09,798::task::1191::Storage.TaskManager.Task::(prepare) Task=`7d982c59-02a7-421c-bd66-13173c216461`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00188026', 'lastCheck': '1.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0004888', 'lastCheck': '8.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00552526', 'lastCheck': '6.0', 'valid': True}} >Thread-69415::DEBUG::2014-09-25 17:02:09,798::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d982c59-02a7-421c-bd66-13173c216461`::moving from state preparing -> state finished >Thread-69415::DEBUG::2014-09-25 17:02:09,799::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69415::DEBUG::2014-09-25 17:02:09,799::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69415::DEBUG::2014-09-25 17:02:09,799::task::993::Storage.TaskManager.Task::(_decref) Task=`7d982c59-02a7-421c-bd66-13173c216461`::ref 0 aborting False >Thread-69415::DEBUG::2014-09-25 17:02:09,801::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:09,824::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:09,826::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69416::DEBUG::2014-09-25 17:02:09,828::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:10,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:10,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148232 s, 69.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:11,304::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:11,305::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:11,305::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:11,305::__init__::232::IOProcess::(_processLogs) (106763) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:11,306::__init__::232::IOProcess::(_processLogs) (106763) Queuing response >Thread-80::DEBUG::2014-09-25 17:02:11,307::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:02:11,316::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000385228 s, 981 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:11,317::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:11,317::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:11,317::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:11,318::__init__::232::IOProcess::(_processLogs) (106764) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:11,318::__init__::232::IOProcess::(_processLogs) (106764) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:11,499::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:11,500::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69417::DEBUG::2014-09-25 17:02:11,500::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69417::DEBUG::2014-09-25 17:02:11,502::task::595::Storage.TaskManager.Task::(_updateState) Task=`c97923ef-a3fc-455d-867f-1c45c2b9f7f1`::moving from state init -> state preparing >Thread-69417::INFO::2014-09-25 17:02:11,502::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69417::INFO::2014-09-25 17:02:11,502::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69417::DEBUG::2014-09-25 17:02:11,503::task::1191::Storage.TaskManager.Task::(prepare) Task=`c97923ef-a3fc-455d-867f-1c45c2b9f7f1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69417::DEBUG::2014-09-25 17:02:11,503::task::595::Storage.TaskManager.Task::(_updateState) Task=`c97923ef-a3fc-455d-867f-1c45c2b9f7f1`::moving from state preparing -> state finished >Thread-69417::DEBUG::2014-09-25 17:02:11,503::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69417::DEBUG::2014-09-25 17:02:11,503::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69417::DEBUG::2014-09-25 17:02:11,504::task::993::Storage.TaskManager.Task::(_decref) Task=`c97923ef-a3fc-455d-867f-1c45c2b9f7f1`::ref 0 aborting False >Thread-69417::DEBUG::2014-09-25 17:02:11,504::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69417::DEBUG::2014-09-25 17:02:11,504::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:11,517::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:11,518::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69418::DEBUG::2014-09-25 17:02:11,518::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69418::DEBUG::2014-09-25 17:02:11,520::task::595::Storage.TaskManager.Task::(_updateState) Task=`0db5a927-3e8f-4e22-b26d-28a4e05155ce`::moving from state init -> state preparing >Thread-69418::INFO::2014-09-25 17:02:11,520::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69418::DEBUG::2014-09-25 17:02:11,521::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`196f9c77-85cb-4684-8ffe-12db06422d6e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69418::DEBUG::2014-09-25 17:02:11,521::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69418::DEBUG::2014-09-25 17:02:11,522::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69418::DEBUG::2014-09-25 17:02:11,522::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`196f9c77-85cb-4684-8ffe-12db06422d6e`::Granted request >Thread-69418::DEBUG::2014-09-25 17:02:11,523::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0db5a927-3e8f-4e22-b26d-28a4e05155ce`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69418::DEBUG::2014-09-25 17:02:11,523::task::993::Storage.TaskManager.Task::(_decref) Task=`0db5a927-3e8f-4e22-b26d-28a4e05155ce`::ref 1 aborting False >Thread-69418::INFO::2014-09-25 17:02:11,527::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69418::DEBUG::2014-09-25 17:02:11,527::task::1191::Storage.TaskManager.Task::(prepare) Task=`0db5a927-3e8f-4e22-b26d-28a4e05155ce`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69418::DEBUG::2014-09-25 17:02:11,528::task::595::Storage.TaskManager.Task::(_updateState) Task=`0db5a927-3e8f-4e22-b26d-28a4e05155ce`::moving from state preparing -> state finished >Thread-69418::DEBUG::2014-09-25 17:02:11,528::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69418::DEBUG::2014-09-25 17:02:11,528::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69418::DEBUG::2014-09-25 17:02:11,529::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69418::DEBUG::2014-09-25 17:02:11,529::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69418::DEBUG::2014-09-25 17:02:11,529::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69418::DEBUG::2014-09-25 17:02:11,529::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69418::DEBUG::2014-09-25 17:02:11,530::task::993::Storage.TaskManager.Task::(_decref) Task=`0db5a927-3e8f-4e22-b26d-28a4e05155ce`::ref 0 aborting False >Thread-69418::DEBUG::2014-09-25 17:02:11,530::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69418::DEBUG::2014-09-25 17:02:11,531::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:12,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:12,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134056 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:12,884::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:12,886::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69419::DEBUG::2014-09-25 17:02:12,888::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:02:13,827::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:13,827::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:13,828::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:02:13,829::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:02:13,829::__init__::232::IOProcess::(_processLogs) (106765) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:13,837::__init__::232::IOProcess::(_processLogs) (106765) Queuing response >Thread-81::DEBUG::2014-09-25 17:02:13,846::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00190503 s, 195 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:13,847::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:13,847::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:13,848::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:13,848::__init__::232::IOProcess::(_processLogs) (106766) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:13,850::__init__::232::IOProcess::(_processLogs) (106766) Queuing response >Dummy-148::DEBUG::2014-09-25 17:02:14,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:14,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146599 s, 69.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:15,905::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:15,906::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69420::DEBUG::2014-09-25 17:02:15,908::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:16,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:16,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0243212 s, 42.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:02:18,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:18,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013301 s, 77.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:18,614::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:18,614::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:18,614::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:18,614::__init__::232::IOProcess::(_processLogs) (106767) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:18,616::__init__::232::IOProcess::(_processLogs) (106767) Queuing response >Thread-79::DEBUG::2014-09-25 17:02:18,617::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:02:18,632::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00249034 s, 226 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:18,633::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:18,634::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:18,634::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:18,634::__init__::232::IOProcess::(_processLogs) (106768) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:18,635::__init__::232::IOProcess::(_processLogs) (106768) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:18,636::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:18,636::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:18,637::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:18,637::__init__::232::IOProcess::(_processLogs) (106769) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:18,638::__init__::232::IOProcess::(_processLogs) (106769) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:18,638::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:18,639::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:18,639::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:18,640::__init__::232::IOProcess::(_processLogs) (106770) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:18,640::__init__::232::IOProcess::(_processLogs) (106770) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:18,640::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:18,641::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:18,642::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:18,642::__init__::232::IOProcess::(_processLogs) (106771) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:18,642::__init__::232::IOProcess::(_processLogs) (106771) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:18,926::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:18,928::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69421::DEBUG::2014-09-25 17:02:18,930::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:20,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:20,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200351 s, 51.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:21,320::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:21,321::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:21,321::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:21,321::__init__::232::IOProcess::(_processLogs) (106772) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:21,322::__init__::232::IOProcess::(_processLogs) (106772) Queuing response >Thread-80::DEBUG::2014-09-25 17:02:21,323::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:02:21,332::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000527419 s, 717 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:21,333::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:21,334::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:21,334::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:21,334::__init__::232::IOProcess::(_processLogs) (106773) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:21,335::__init__::232::IOProcess::(_processLogs) (106773) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:21,561::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:21,562::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69422::DEBUG::2014-09-25 17:02:21,562::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69422::DEBUG::2014-09-25 17:02:21,564::task::595::Storage.TaskManager.Task::(_updateState) Task=`37c43e26-45a5-4baf-a86a-c234023c8494`::moving from state init -> state preparing >Thread-69422::INFO::2014-09-25 17:02:21,565::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69422::INFO::2014-09-25 17:02:21,565::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69422::DEBUG::2014-09-25 17:02:21,565::task::1191::Storage.TaskManager.Task::(prepare) Task=`37c43e26-45a5-4baf-a86a-c234023c8494`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69422::DEBUG::2014-09-25 17:02:21,566::task::595::Storage.TaskManager.Task::(_updateState) Task=`37c43e26-45a5-4baf-a86a-c234023c8494`::moving from state preparing -> state finished >Thread-69422::DEBUG::2014-09-25 17:02:21,566::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69422::DEBUG::2014-09-25 17:02:21,566::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69422::DEBUG::2014-09-25 17:02:21,566::task::993::Storage.TaskManager.Task::(_decref) Task=`37c43e26-45a5-4baf-a86a-c234023c8494`::ref 0 aborting False >Thread-69422::DEBUG::2014-09-25 17:02:21,567::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69422::DEBUG::2014-09-25 17:02:21,567::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:21,581::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69423::DEBUG::2014-09-25 17:02:21,582::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:02:21,582::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69423::DEBUG::2014-09-25 17:02:21,584::task::595::Storage.TaskManager.Task::(_updateState) Task=`9b32e068-6201-4719-8c6e-9051f22f193f`::moving from state init -> state preparing >Thread-69423::INFO::2014-09-25 17:02:21,585::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69423::DEBUG::2014-09-25 17:02:21,585::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f7e79286-2144-4dbf-ba59-00e38decefac`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69423::DEBUG::2014-09-25 17:02:21,586::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69423::DEBUG::2014-09-25 17:02:21,586::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69423::DEBUG::2014-09-25 17:02:21,586::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f7e79286-2144-4dbf-ba59-00e38decefac`::Granted request >Thread-69423::DEBUG::2014-09-25 17:02:21,587::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9b32e068-6201-4719-8c6e-9051f22f193f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69423::DEBUG::2014-09-25 17:02:21,587::task::993::Storage.TaskManager.Task::(_decref) Task=`9b32e068-6201-4719-8c6e-9051f22f193f`::ref 1 aborting False >Thread-69423::INFO::2014-09-25 17:02:21,599::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69423::DEBUG::2014-09-25 17:02:21,600::task::1191::Storage.TaskManager.Task::(prepare) Task=`9b32e068-6201-4719-8c6e-9051f22f193f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69423::DEBUG::2014-09-25 17:02:21,601::task::595::Storage.TaskManager.Task::(_updateState) Task=`9b32e068-6201-4719-8c6e-9051f22f193f`::moving from state preparing -> state finished >Thread-69423::DEBUG::2014-09-25 17:02:21,602::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69423::DEBUG::2014-09-25 17:02:21,602::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69423::DEBUG::2014-09-25 17:02:21,604::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69423::DEBUG::2014-09-25 17:02:21,604::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69423::DEBUG::2014-09-25 17:02:21,605::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69423::DEBUG::2014-09-25 17:02:21,606::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69423::DEBUG::2014-09-25 17:02:21,608::task::993::Storage.TaskManager.Task::(_decref) Task=`9b32e068-6201-4719-8c6e-9051f22f193f`::ref 0 aborting False >Thread-69423::DEBUG::2014-09-25 17:02:21,610::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69423::DEBUG::2014-09-25 17:02:21,610::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:21,946::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:21,948::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69424::DEBUG::2014-09-25 17:02:21,949::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:22,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:22,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152692 s, 67.1 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 17:02:23,852::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:02:23,857::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:23,857::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:23,858::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:23,858::__init__::232::IOProcess::(_processLogs) (106774) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:02:23,859::__init__::232::IOProcess::(_processLogs) (106774) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:23,860::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:23,862::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:23,864::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:23,865::__init__::232::IOProcess::(_processLogs) (106775) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:02:23,868::__init__::232::IOProcess::(_processLogs) (106775) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:23,870::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:23,872::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-81::DEBUG::2014-09-25 17:02:23,872::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:02:23,873::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:23,874::__init__::232::IOProcess::(_processLogs) (106776) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:02:23,875::__init__::232::IOProcess::(_processLogs) (106776) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:23,876::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:23,876::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:23,876::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:23,876::__init__::232::IOProcess::(_processLogs) (106777) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:02:23,905::__init__::232::IOProcess::(_processLogs) (106777) Queuing response >Thread-81::DEBUG::2014-09-25 17:02:23,905::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:02:23,906::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:23,907::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:23,907::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:23,907::__init__::232::IOProcess::(_processLogs) (106778) Got request for method 'readfile' >Thread-81::DEBUG::2014-09-25 17:02:23,915::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:02:23,915::__init__::232::IOProcess::(_processLogs) (106778) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:23,917::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:23,917::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:23,918::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:23,918::__init__::232::IOProcess::(_processLogs) (106779) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:23,918::__init__::232::IOProcess::(_processLogs) (106779) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:23,919::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:23,919::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:23,919::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:23,920::__init__::232::IOProcess::(_processLogs) (106780) Got request for method 'glob' >Thread-81::DEBUG::2014-09-25 17:02:23,924::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:02:23,924::__init__::232::IOProcess::(_processLogs) (106780) Queuing response >Thread-81::INFO::2014-09-25 17:02:23,925::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:02:23,926::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:02:23,927::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:23,927::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:23,927::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:23,927::__init__::232::IOProcess::(_processLogs) (106781) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:23,928::__init__::232::IOProcess::(_processLogs) (106781) Queuing response >Thread-81::DEBUG::2014-09-25 17:02:23,929::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:02:23,942::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00284202 s, 131 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:23,943::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:23,943::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:23,943::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:23,944::__init__::232::IOProcess::(_processLogs) (106782) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:23,945::__init__::232::IOProcess::(_processLogs) (106782) Queuing response >Dummy-148::DEBUG::2014-09-25 17:02:24,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:24,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181459 s, 56.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:24,963::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:24,965::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69428::DEBUG::2014-09-25 17:02:24,968::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ab97a50-d63d-47b5-b603-b193a39b5973`::moving from state init -> state preparing >Thread-69428::INFO::2014-09-25 17:02:24,968::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69428::INFO::2014-09-25 17:02:24,968::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00249034', 'lastCheck': '6.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000527419', 'lastCheck': '3.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00284202', 'lastCheck': '1.0', 'valid': True}} >Thread-69428::DEBUG::2014-09-25 17:02:24,969::task::1191::Storage.TaskManager.Task::(prepare) Task=`0ab97a50-d63d-47b5-b603-b193a39b5973`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00249034', 'lastCheck': '6.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000527419', 'lastCheck': '3.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00284202', 'lastCheck': '1.0', 'valid': True}} >Thread-69428::DEBUG::2014-09-25 17:02:24,969::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ab97a50-d63d-47b5-b603-b193a39b5973`::moving from state preparing -> state finished >Thread-69428::DEBUG::2014-09-25 17:02:24,969::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69428::DEBUG::2014-09-25 17:02:24,969::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69428::DEBUG::2014-09-25 17:02:24,970::task::993::Storage.TaskManager.Task::(_decref) Task=`0ab97a50-d63d-47b5-b603-b193a39b5973`::ref 0 aborting False >Thread-69428::DEBUG::2014-09-25 17:02:24,972::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:24,993::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:24,994::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69429::DEBUG::2014-09-25 17:02:24,996::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:26,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:26,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158426 s, 64.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:28,058::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:28,059::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69430::DEBUG::2014-09-25 17:02:28,061::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:02:28,642::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:28,643::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:28,643::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:28,643::__init__::232::IOProcess::(_processLogs) (106783) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:28,644::__init__::232::IOProcess::(_processLogs) (106783) Queuing response >Thread-79::DEBUG::2014-09-25 17:02:28,645::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:02:28,654::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000499089 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:28,655::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:28,655::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:28,655::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:28,656::__init__::232::IOProcess::(_processLogs) (106784) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:28,656::__init__::232::IOProcess::(_processLogs) (106784) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:28,658::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:28,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:28,658::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:28,659::__init__::232::IOProcess::(_processLogs) (106785) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:28,659::__init__::232::IOProcess::(_processLogs) (106785) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:28,660::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:28,660::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:28,661::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:28,661::__init__::232::IOProcess::(_processLogs) (106786) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:28,661::__init__::232::IOProcess::(_processLogs) (106786) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:28,662::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:28,662::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:28,663::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:28,664::__init__::232::IOProcess::(_processLogs) (106787) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:28,664::__init__::232::IOProcess::(_processLogs) (106787) Queuing response >Dummy-148::DEBUG::2014-09-25 17:02:28,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:28,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133209 s, 76.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:02:30,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:30,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133037 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:31,079::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:31,080::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69431::DEBUG::2014-09-25 17:02:31,082::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:02:31,337::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:31,337::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:31,338::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:31,338::__init__::232::IOProcess::(_processLogs) (106788) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:31,338::__init__::232::IOProcess::(_processLogs) (106788) Queuing response >Thread-80::DEBUG::2014-09-25 17:02:31,340::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:02:31,348::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000399482 s, 946 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:31,349::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:31,350::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:31,350::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:31,350::__init__::232::IOProcess::(_processLogs) (106789) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:31,351::__init__::232::IOProcess::(_processLogs) (106789) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:31,639::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69432::DEBUG::2014-09-25 17:02:31,643::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:02:31,643::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69432::DEBUG::2014-09-25 17:02:31,649::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c0a5917-c951-44ce-873d-fc626472067f`::moving from state init -> state preparing >Thread-69432::INFO::2014-09-25 17:02:31,650::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69432::INFO::2014-09-25 17:02:31,652::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69432::DEBUG::2014-09-25 17:02:31,653::task::1191::Storage.TaskManager.Task::(prepare) Task=`1c0a5917-c951-44ce-873d-fc626472067f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69432::DEBUG::2014-09-25 17:02:31,653::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c0a5917-c951-44ce-873d-fc626472067f`::moving from state preparing -> state finished >Thread-69432::DEBUG::2014-09-25 17:02:31,654::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69432::DEBUG::2014-09-25 17:02:31,655::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69432::DEBUG::2014-09-25 17:02:31,656::task::993::Storage.TaskManager.Task::(_decref) Task=`1c0a5917-c951-44ce-873d-fc626472067f`::ref 0 aborting False >Thread-69432::DEBUG::2014-09-25 17:02:31,657::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69432::DEBUG::2014-09-25 17:02:31,659::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:31,672::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:31,674::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69433::DEBUG::2014-09-25 17:02:31,674::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69433::DEBUG::2014-09-25 17:02:31,675::task::595::Storage.TaskManager.Task::(_updateState) Task=`0bbcb7e0-aee2-4324-b1be-b9dd5e36a2e8`::moving from state init -> state preparing >Thread-69433::INFO::2014-09-25 17:02:31,676::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69433::DEBUG::2014-09-25 17:02:31,676::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`eb85bf98-034e-450a-8ae4-764f247ece27`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69433::DEBUG::2014-09-25 17:02:31,677::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69433::DEBUG::2014-09-25 17:02:31,677::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69433::DEBUG::2014-09-25 17:02:31,677::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`eb85bf98-034e-450a-8ae4-764f247ece27`::Granted request >Thread-69433::DEBUG::2014-09-25 17:02:31,678::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0bbcb7e0-aee2-4324-b1be-b9dd5e36a2e8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69433::DEBUG::2014-09-25 17:02:31,678::task::993::Storage.TaskManager.Task::(_decref) Task=`0bbcb7e0-aee2-4324-b1be-b9dd5e36a2e8`::ref 1 aborting False >Thread-69433::INFO::2014-09-25 17:02:31,682::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69433::DEBUG::2014-09-25 17:02:31,683::task::1191::Storage.TaskManager.Task::(prepare) Task=`0bbcb7e0-aee2-4324-b1be-b9dd5e36a2e8`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69433::DEBUG::2014-09-25 17:02:31,683::task::595::Storage.TaskManager.Task::(_updateState) Task=`0bbcb7e0-aee2-4324-b1be-b9dd5e36a2e8`::moving from state preparing -> state finished >Thread-69433::DEBUG::2014-09-25 17:02:31,683::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69433::DEBUG::2014-09-25 17:02:31,683::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69433::DEBUG::2014-09-25 17:02:31,684::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69433::DEBUG::2014-09-25 17:02:31,684::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69433::DEBUG::2014-09-25 17:02:31,684::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69433::DEBUG::2014-09-25 17:02:31,685::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69433::DEBUG::2014-09-25 17:02:31,685::task::993::Storage.TaskManager.Task::(_decref) Task=`0bbcb7e0-aee2-4324-b1be-b9dd5e36a2e8`::ref 0 aborting False >Thread-69433::DEBUG::2014-09-25 17:02:31,685::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69433::DEBUG::2014-09-25 17:02:31,686::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:32,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:32,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133869 s, 76.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:33,947::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:33,947::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:33,948::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:33,948::__init__::232::IOProcess::(_processLogs) (106790) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:33,948::__init__::232::IOProcess::(_processLogs) (106790) Queuing response >Thread-81::DEBUG::2014-09-25 17:02:33,949::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:02:33,965::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00330036 s, 112 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:33,966::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:33,967::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:33,967::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:33,967::__init__::232::IOProcess::(_processLogs) (106791) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:33,971::__init__::232::IOProcess::(_processLogs) (106791) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:34,099::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:34,101::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69434::DEBUG::2014-09-25 17:02:34,103::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:34,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:34,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155275 s, 65.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:02:36,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:36,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212058 s, 48.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:37,119::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:37,120::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69435::DEBUG::2014-09-25 17:02:37,122::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:02:38,664::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:38,665::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:38,665::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:38,665::__init__::232::IOProcess::(_processLogs) (106792) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:38,671::__init__::232::IOProcess::(_processLogs) (106792) Queuing response >Thread-79::DEBUG::2014-09-25 17:02:38,673::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:02:38,689::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000396427 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:38,690::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:38,690::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:38,691::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:38,691::__init__::232::IOProcess::(_processLogs) (106793) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:38,691::__init__::232::IOProcess::(_processLogs) (106793) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:38,692::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:38,693::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:38,693::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:38,693::__init__::232::IOProcess::(_processLogs) (106794) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:38,694::__init__::232::IOProcess::(_processLogs) (106794) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:38,696::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:38,696::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:38,696::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:38,697::__init__::232::IOProcess::(_processLogs) (106795) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:38,697::__init__::232::IOProcess::(_processLogs) (106795) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:38,699::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:38,699::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:38,699::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:38,700::__init__::232::IOProcess::(_processLogs) (106796) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:38,700::__init__::232::IOProcess::(_processLogs) (106796) Queuing response >Dummy-148::DEBUG::2014-09-25 17:02:38,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:38,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018067 s, 56.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:40,136::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:40,137::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69436::DEBUG::2014-09-25 17:02:40,140::task::595::Storage.TaskManager.Task::(_updateState) Task=`237ea85c-eb87-405a-a8ae-6abb33ee9391`::moving from state init -> state preparing >Thread-69436::INFO::2014-09-25 17:02:40,140::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69436::INFO::2014-09-25 17:02:40,141::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000396427', 'lastCheck': '1.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000399482', 'lastCheck': '8.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00330036', 'lastCheck': '6.2', 'valid': True}} >Thread-69436::DEBUG::2014-09-25 17:02:40,141::task::1191::Storage.TaskManager.Task::(prepare) Task=`237ea85c-eb87-405a-a8ae-6abb33ee9391`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000396427', 'lastCheck': '1.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000399482', 'lastCheck': '8.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00330036', 'lastCheck': '6.2', 'valid': True}} >Thread-69436::DEBUG::2014-09-25 17:02:40,141::task::595::Storage.TaskManager.Task::(_updateState) Task=`237ea85c-eb87-405a-a8ae-6abb33ee9391`::moving from state preparing -> state finished >Thread-69436::DEBUG::2014-09-25 17:02:40,141::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69436::DEBUG::2014-09-25 17:02:40,142::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69436::DEBUG::2014-09-25 17:02:40,142::task::993::Storage.TaskManager.Task::(_decref) Task=`237ea85c-eb87-405a-a8ae-6abb33ee9391`::ref 0 aborting False >Thread-69436::DEBUG::2014-09-25 17:02:40,144::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:40,165::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:40,167::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69437::DEBUG::2014-09-25 17:02:40,169::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:40,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:40,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164558 s, 62.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:41,353::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:41,353::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:41,354::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:41,354::__init__::232::IOProcess::(_processLogs) (106797) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:41,354::__init__::232::IOProcess::(_processLogs) (106797) Queuing response >Thread-80::DEBUG::2014-09-25 17:02:41,355::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:02:41,364::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000444074 s, 851 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:41,366::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:41,366::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:41,366::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:41,366::__init__::232::IOProcess::(_processLogs) (106798) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:41,367::__init__::232::IOProcess::(_processLogs) (106798) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:41,713::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:41,714::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69438::DEBUG::2014-09-25 17:02:41,714::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69438::DEBUG::2014-09-25 17:02:41,716::task::595::Storage.TaskManager.Task::(_updateState) Task=`996e01c5-6251-4f13-8042-25872b612bca`::moving from state init -> state preparing >Thread-69438::INFO::2014-09-25 17:02:41,716::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69438::INFO::2014-09-25 17:02:41,717::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69438::DEBUG::2014-09-25 17:02:41,717::task::1191::Storage.TaskManager.Task::(prepare) Task=`996e01c5-6251-4f13-8042-25872b612bca`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69438::DEBUG::2014-09-25 17:02:41,717::task::595::Storage.TaskManager.Task::(_updateState) Task=`996e01c5-6251-4f13-8042-25872b612bca`::moving from state preparing -> state finished >Thread-69438::DEBUG::2014-09-25 17:02:41,718::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69438::DEBUG::2014-09-25 17:02:41,718::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69438::DEBUG::2014-09-25 17:02:41,718::task::993::Storage.TaskManager.Task::(_decref) Task=`996e01c5-6251-4f13-8042-25872b612bca`::ref 0 aborting False >Thread-69438::DEBUG::2014-09-25 17:02:41,718::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69438::DEBUG::2014-09-25 17:02:41,719::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:41,732::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:41,733::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69439::DEBUG::2014-09-25 17:02:41,733::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69439::DEBUG::2014-09-25 17:02:41,735::task::595::Storage.TaskManager.Task::(_updateState) Task=`23ef7808-4427-4fc1-ba6e-6aaefe9c8cf6`::moving from state init -> state preparing >Thread-69439::INFO::2014-09-25 17:02:41,735::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69439::DEBUG::2014-09-25 17:02:41,736::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9107c31d-f919-45ef-9f2c-e642ad5c3fd2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69439::DEBUG::2014-09-25 17:02:41,736::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69439::DEBUG::2014-09-25 17:02:41,737::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69439::DEBUG::2014-09-25 17:02:41,737::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9107c31d-f919-45ef-9f2c-e642ad5c3fd2`::Granted request >Thread-69439::DEBUG::2014-09-25 17:02:41,737::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`23ef7808-4427-4fc1-ba6e-6aaefe9c8cf6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69439::DEBUG::2014-09-25 17:02:41,738::task::993::Storage.TaskManager.Task::(_decref) Task=`23ef7808-4427-4fc1-ba6e-6aaefe9c8cf6`::ref 1 aborting False >Thread-69439::INFO::2014-09-25 17:02:41,742::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69439::DEBUG::2014-09-25 17:02:41,742::task::1191::Storage.TaskManager.Task::(prepare) Task=`23ef7808-4427-4fc1-ba6e-6aaefe9c8cf6`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69439::DEBUG::2014-09-25 17:02:41,743::task::595::Storage.TaskManager.Task::(_updateState) Task=`23ef7808-4427-4fc1-ba6e-6aaefe9c8cf6`::moving from state preparing -> state finished >Thread-69439::DEBUG::2014-09-25 17:02:41,743::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69439::DEBUG::2014-09-25 17:02:41,743::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69439::DEBUG::2014-09-25 17:02:41,743::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69439::DEBUG::2014-09-25 17:02:41,744::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69439::DEBUG::2014-09-25 17:02:41,744::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69439::DEBUG::2014-09-25 17:02:41,744::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69439::DEBUG::2014-09-25 17:02:41,745::task::993::Storage.TaskManager.Task::(_decref) Task=`23ef7808-4427-4fc1-ba6e-6aaefe9c8cf6`::ref 0 aborting False >Thread-69439::DEBUG::2014-09-25 17:02:41,745::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69439::DEBUG::2014-09-25 17:02:41,745::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:43,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:43,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157746 s, 64.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:43,228::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:43,229::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69440::DEBUG::2014-09-25 17:02:43,231::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:02:43,973::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:43,973::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:43,974::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:43,974::__init__::232::IOProcess::(_processLogs) (106799) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:43,978::__init__::232::IOProcess::(_processLogs) (106799) Queuing response >Thread-81::DEBUG::2014-09-25 17:02:43,979::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:02:43,993::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00324348 s, 114 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:43,994::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:43,994::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:43,994::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:43,994::__init__::232::IOProcess::(_processLogs) (106800) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:43,996::__init__::232::IOProcess::(_processLogs) (106800) Queuing response >Dummy-148::DEBUG::2014-09-25 17:02:45,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:45,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171216 s, 59.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:46,248::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:46,251::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69441::DEBUG::2014-09-25 17:02:46,258::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:47,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:47,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0120737 s, 84.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:48,702::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:48,702::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:48,702::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:48,703::__init__::232::IOProcess::(_processLogs) (106801) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:48,703::__init__::232::IOProcess::(_processLogs) (106801) Queuing response >Thread-79::DEBUG::2014-09-25 17:02:48,704::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:02:48,714::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000406292 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:48,715::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:48,715::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:48,715::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:48,716::__init__::232::IOProcess::(_processLogs) (106802) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:48,716::__init__::232::IOProcess::(_processLogs) (106802) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:48,718::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:48,718::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:48,719::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:48,719::__init__::232::IOProcess::(_processLogs) (106803) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:48,719::__init__::232::IOProcess::(_processLogs) (106803) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:48,720::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:48,720::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:48,720::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:48,720::__init__::232::IOProcess::(_processLogs) (106804) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:48,721::__init__::232::IOProcess::(_processLogs) (106804) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:48,721::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:48,722::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:48,722::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:48,722::__init__::232::IOProcess::(_processLogs) (106805) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:48,722::__init__::232::IOProcess::(_processLogs) (106805) Queuing response >Dummy-148::DEBUG::2014-09-25 17:02:49,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:49,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193169 s, 53.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:49,278::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:49,279::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69442::DEBUG::2014-09-25 17:02:49,281::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:51,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:51,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0426032 s, 24.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:51,370::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:51,370::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:51,370::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:51,372::__init__::232::IOProcess::(_processLogs) (106806) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:02:51,372::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:02:51,373::__init__::232::IOProcess::(_processLogs) (106806) Queuing response >Thread-80::DEBUG::2014-09-25 17:02:51,382::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000579492 s, 652 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:51,383::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:51,383::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:51,383::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:51,384::__init__::232::IOProcess::(_processLogs) (106807) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:51,384::__init__::232::IOProcess::(_processLogs) (106807) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:51,778::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69443::DEBUG::2014-09-25 17:02:51,779::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:02:51,779::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69443::DEBUG::2014-09-25 17:02:51,781::task::595::Storage.TaskManager.Task::(_updateState) Task=`942d8442-4cee-41a4-a520-5b78ff72d366`::moving from state init -> state preparing >Thread-69443::INFO::2014-09-25 17:02:51,781::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69443::INFO::2014-09-25 17:02:51,782::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69443::DEBUG::2014-09-25 17:02:51,782::task::1191::Storage.TaskManager.Task::(prepare) Task=`942d8442-4cee-41a4-a520-5b78ff72d366`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69443::DEBUG::2014-09-25 17:02:51,782::task::595::Storage.TaskManager.Task::(_updateState) Task=`942d8442-4cee-41a4-a520-5b78ff72d366`::moving from state preparing -> state finished >Thread-69443::DEBUG::2014-09-25 17:02:51,783::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69443::DEBUG::2014-09-25 17:02:51,783::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69443::DEBUG::2014-09-25 17:02:51,783::task::993::Storage.TaskManager.Task::(_decref) Task=`942d8442-4cee-41a4-a520-5b78ff72d366`::ref 0 aborting False >Thread-69443::DEBUG::2014-09-25 17:02:51,783::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69443::DEBUG::2014-09-25 17:02:51,784::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:51,798::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:51,799::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69444::DEBUG::2014-09-25 17:02:51,799::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69444::DEBUG::2014-09-25 17:02:51,801::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d843d82-d198-4731-b448-5239632e4d46`::moving from state init -> state preparing >Thread-69444::INFO::2014-09-25 17:02:51,801::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69444::DEBUG::2014-09-25 17:02:51,802::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ddbf7f74-e6dc-4211-bba8-e23c5c8b1105`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69444::DEBUG::2014-09-25 17:02:51,802::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69444::DEBUG::2014-09-25 17:02:51,802::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69444::DEBUG::2014-09-25 17:02:51,802::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ddbf7f74-e6dc-4211-bba8-e23c5c8b1105`::Granted request >Thread-69444::DEBUG::2014-09-25 17:02:51,803::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2d843d82-d198-4731-b448-5239632e4d46`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69444::DEBUG::2014-09-25 17:02:51,803::task::993::Storage.TaskManager.Task::(_decref) Task=`2d843d82-d198-4731-b448-5239632e4d46`::ref 1 aborting False >Thread-69444::INFO::2014-09-25 17:02:51,808::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69444::DEBUG::2014-09-25 17:02:51,808::task::1191::Storage.TaskManager.Task::(prepare) Task=`2d843d82-d198-4731-b448-5239632e4d46`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69444::DEBUG::2014-09-25 17:02:51,808::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d843d82-d198-4731-b448-5239632e4d46`::moving from state preparing -> state finished >Thread-69444::DEBUG::2014-09-25 17:02:51,808::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69444::DEBUG::2014-09-25 17:02:51,809::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69444::DEBUG::2014-09-25 17:02:51,809::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69444::DEBUG::2014-09-25 17:02:51,809::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69444::DEBUG::2014-09-25 17:02:51,809::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69444::DEBUG::2014-09-25 17:02:51,810::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69444::DEBUG::2014-09-25 17:02:51,810::task::993::Storage.TaskManager.Task::(_decref) Task=`2d843d82-d198-4731-b448-5239632e4d46`::ref 0 aborting False >Thread-69444::DEBUG::2014-09-25 17:02:51,810::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69444::DEBUG::2014-09-25 17:02:51,811::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:52,298::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:52,299::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69445::DEBUG::2014-09-25 17:02:52,301::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:53,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:53,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203917 s, 50.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:53,997::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:53,998::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:53,998::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:53,998::__init__::232::IOProcess::(_processLogs) (106808) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:54,006::__init__::232::IOProcess::(_processLogs) (106808) Queuing response >Thread-81::DEBUG::2014-09-25 17:02:54,007::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:02:54,024::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00327662 s, 113 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:54,025::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:54,025::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:54,026::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:54,026::__init__::232::IOProcess::(_processLogs) (106809) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:54,027::__init__::232::IOProcess::(_processLogs) (106809) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:55,314::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:55,316::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69446::DEBUG::2014-09-25 17:02:55,318::task::595::Storage.TaskManager.Task::(_updateState) Task=`9ad40851-dcd1-4313-8cea-69d41ad775fb`::moving from state init -> state preparing >Thread-69446::INFO::2014-09-25 17:02:55,319::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69446::INFO::2014-09-25 17:02:55,319::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000406292', 'lastCheck': '6.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000579492', 'lastCheck': '3.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00327662', 'lastCheck': '1.3', 'valid': True}} >Thread-69446::DEBUG::2014-09-25 17:02:55,319::task::1191::Storage.TaskManager.Task::(prepare) Task=`9ad40851-dcd1-4313-8cea-69d41ad775fb`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000406292', 'lastCheck': '6.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000579492', 'lastCheck': '3.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00327662', 'lastCheck': '1.3', 'valid': True}} >Thread-69446::DEBUG::2014-09-25 17:02:55,320::task::595::Storage.TaskManager.Task::(_updateState) Task=`9ad40851-dcd1-4313-8cea-69d41ad775fb`::moving from state preparing -> state finished >Thread-69446::DEBUG::2014-09-25 17:02:55,320::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69446::DEBUG::2014-09-25 17:02:55,320::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69446::DEBUG::2014-09-25 17:02:55,320::task::993::Storage.TaskManager.Task::(_decref) Task=`9ad40851-dcd1-4313-8cea-69d41ad775fb`::ref 0 aborting False >Dummy-148::DEBUG::2014-09-25 17:02:55,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-69446::DEBUG::2014-09-25 17:02:55,332::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:55,352::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:55,354::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69447::DEBUG::2014-09-25 17:02:55,356::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:02:55,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0231487 s, 44.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:02:57,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:57,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202414 s, 50.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:02:58,424::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:02:58,425::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69448::DEBUG::2014-09-25 17:02:58,427::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:02:58,724::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:58,724::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:58,724::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:58,725::__init__::232::IOProcess::(_processLogs) (106810) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:58,725::__init__::232::IOProcess::(_processLogs) (106810) Queuing response >Thread-79::DEBUG::2014-09-25 17:02:58,726::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:02:58,735::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000432466 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:02:58,736::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:58,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:58,737::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:58,737::__init__::232::IOProcess::(_processLogs) (106811) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:02:58,738::__init__::232::IOProcess::(_processLogs) (106811) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:58,739::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:58,740::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:58,740::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:58,740::__init__::232::IOProcess::(_processLogs) (106812) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:58,741::__init__::232::IOProcess::(_processLogs) (106812) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:58,741::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:58,742::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:58,742::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:58,743::__init__::232::IOProcess::(_processLogs) (106813) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:58,743::__init__::232::IOProcess::(_processLogs) (106813) Queuing response >Thread-29::DEBUG::2014-09-25 17:02:58,743::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:02:58,744::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:02:58,745::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:02:58,745::__init__::232::IOProcess::(_processLogs) (106814) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:02:58,745::__init__::232::IOProcess::(_processLogs) (106814) Queuing response >Dummy-148::DEBUG::2014-09-25 17:02:59,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:02:59,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164358 s, 62.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:01,387::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:01,387::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:01,387::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:01,388::__init__::232::IOProcess::(_processLogs) (106815) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:01,388::__init__::232::IOProcess::(_processLogs) (106815) Queuing response >Thread-80::DEBUG::2014-09-25 17:03:01,389::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:03:01,399::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000454821 s, 831 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:01,400::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:01,400::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:01,400::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:01,400::__init__::232::IOProcess::(_processLogs) (106816) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:01,401::__init__::232::IOProcess::(_processLogs) (106816) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:01,443::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:01,444::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69449::DEBUG::2014-09-25 17:03:01,446::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:01,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:01,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171811 s, 59.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:01,839::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:01,841::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69450::DEBUG::2014-09-25 17:03:01,841::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69450::DEBUG::2014-09-25 17:03:01,843::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb1121a5-0f89-4137-ad65-7c4dd781a0c3`::moving from state init -> state preparing >Thread-69450::INFO::2014-09-25 17:03:01,843::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69450::INFO::2014-09-25 17:03:01,844::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69450::DEBUG::2014-09-25 17:03:01,844::task::1191::Storage.TaskManager.Task::(prepare) Task=`fb1121a5-0f89-4137-ad65-7c4dd781a0c3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69450::DEBUG::2014-09-25 17:03:01,844::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb1121a5-0f89-4137-ad65-7c4dd781a0c3`::moving from state preparing -> state finished >Thread-69450::DEBUG::2014-09-25 17:03:01,844::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69450::DEBUG::2014-09-25 17:03:01,845::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69450::DEBUG::2014-09-25 17:03:01,845::task::993::Storage.TaskManager.Task::(_decref) Task=`fb1121a5-0f89-4137-ad65-7c4dd781a0c3`::ref 0 aborting False >Thread-69450::DEBUG::2014-09-25 17:03:01,845::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69450::DEBUG::2014-09-25 17:03:01,846::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:01,862::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:01,863::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69451::DEBUG::2014-09-25 17:03:01,863::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69451::DEBUG::2014-09-25 17:03:01,864::task::595::Storage.TaskManager.Task::(_updateState) Task=`5a564383-2f5b-4654-be63-9f63456b12f5`::moving from state init -> state preparing >Thread-69451::INFO::2014-09-25 17:03:01,865::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69451::DEBUG::2014-09-25 17:03:01,865::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`470b0494-a912-4525-923e-4430dcf4abd5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69451::DEBUG::2014-09-25 17:03:01,866::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69451::DEBUG::2014-09-25 17:03:01,866::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69451::DEBUG::2014-09-25 17:03:01,866::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`470b0494-a912-4525-923e-4430dcf4abd5`::Granted request >Thread-69451::DEBUG::2014-09-25 17:03:01,868::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5a564383-2f5b-4654-be63-9f63456b12f5`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69451::DEBUG::2014-09-25 17:03:01,868::task::993::Storage.TaskManager.Task::(_decref) Task=`5a564383-2f5b-4654-be63-9f63456b12f5`::ref 1 aborting False >Thread-69451::INFO::2014-09-25 17:03:01,873::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69451::DEBUG::2014-09-25 17:03:01,873::task::1191::Storage.TaskManager.Task::(prepare) Task=`5a564383-2f5b-4654-be63-9f63456b12f5`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69451::DEBUG::2014-09-25 17:03:01,873::task::595::Storage.TaskManager.Task::(_updateState) Task=`5a564383-2f5b-4654-be63-9f63456b12f5`::moving from state preparing -> state finished >Thread-69451::DEBUG::2014-09-25 17:03:01,873::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69451::DEBUG::2014-09-25 17:03:01,874::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69451::DEBUG::2014-09-25 17:03:01,874::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69451::DEBUG::2014-09-25 17:03:01,874::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69451::DEBUG::2014-09-25 17:03:01,875::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69451::DEBUG::2014-09-25 17:03:01,875::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69451::DEBUG::2014-09-25 17:03:01,875::task::993::Storage.TaskManager.Task::(_decref) Task=`5a564383-2f5b-4654-be63-9f63456b12f5`::ref 0 aborting False >Thread-69451::DEBUG::2014-09-25 17:03:01,876::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69451::DEBUG::2014-09-25 17:03:01,876::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:03,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:03,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0195046 s, 52.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:04,029::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:04,029::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:04,030::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:04,030::__init__::232::IOProcess::(_processLogs) (106817) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:04,036::__init__::232::IOProcess::(_processLogs) (106817) Queuing response >Thread-81::DEBUG::2014-09-25 17:03:04,037::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:03:04,052::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00729324 s, 50.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:04,054::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:04,054::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:04,054::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:04,055::__init__::232::IOProcess::(_processLogs) (106818) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:04,063::__init__::232::IOProcess::(_processLogs) (106818) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:04,461::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:04,462::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69452::DEBUG::2014-09-25 17:03:04,464::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:05,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:05,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150217 s, 68.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:07,482::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:07,483::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69453::DEBUG::2014-09-25 17:03:07,485::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:07,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:07,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181171 s, 56.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:08,745::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:08,746::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:08,746::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:08,746::__init__::232::IOProcess::(_processLogs) (106819) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:08,758::__init__::232::IOProcess::(_processLogs) (106819) Queuing response >Thread-79::DEBUG::2014-09-25 17:03:08,759::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:03:08,791::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00698352 s, 80.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:08,792::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:08,793::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:08,793::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:08,793::__init__::232::IOProcess::(_processLogs) (106820) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:08,799::__init__::232::IOProcess::(_processLogs) (106820) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:08,800::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:08,801::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:08,801::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:08,802::__init__::232::IOProcess::(_processLogs) (106821) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:08,802::__init__::232::IOProcess::(_processLogs) (106821) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:08,802::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:08,803::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:08,803::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:08,804::__init__::232::IOProcess::(_processLogs) (106822) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:08,804::__init__::232::IOProcess::(_processLogs) (106822) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:08,804::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:08,805::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:08,806::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:08,806::__init__::232::IOProcess::(_processLogs) (106823) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:08,806::__init__::232::IOProcess::(_processLogs) (106823) Queuing response >Dummy-148::DEBUG::2014-09-25 17:03:09,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:09,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171205 s, 59.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:10,502::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:10,503::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69454::DEBUG::2014-09-25 17:03:10,506::task::595::Storage.TaskManager.Task::(_updateState) Task=`43693cc4-79e9-4241-a19b-2d68c430bcdf`::moving from state init -> state preparing >Thread-69454::INFO::2014-09-25 17:03:10,506::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69454::INFO::2014-09-25 17:03:10,507::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00698352', 'lastCheck': '1.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000454821', 'lastCheck': '9.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00729324', 'lastCheck': '6.4', 'valid': True}} >Thread-69454::DEBUG::2014-09-25 17:03:10,507::task::1191::Storage.TaskManager.Task::(prepare) Task=`43693cc4-79e9-4241-a19b-2d68c430bcdf`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00698352', 'lastCheck': '1.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000454821', 'lastCheck': '9.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00729324', 'lastCheck': '6.4', 'valid': True}} >Thread-69454::DEBUG::2014-09-25 17:03:10,507::task::595::Storage.TaskManager.Task::(_updateState) Task=`43693cc4-79e9-4241-a19b-2d68c430bcdf`::moving from state preparing -> state finished >Thread-69454::DEBUG::2014-09-25 17:03:10,507::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69454::DEBUG::2014-09-25 17:03:10,508::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69454::DEBUG::2014-09-25 17:03:10,508::task::993::Storage.TaskManager.Task::(_decref) Task=`43693cc4-79e9-4241-a19b-2d68c430bcdf`::ref 0 aborting False >Thread-69454::DEBUG::2014-09-25 17:03:10,510::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:10,532::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:10,533::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69455::DEBUG::2014-09-25 17:03:10,535::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:03:11,403::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:11,403::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:11,404::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:11,404::__init__::232::IOProcess::(_processLogs) (106824) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:11,404::__init__::232::IOProcess::(_processLogs) (106824) Queuing response >Thread-80::DEBUG::2014-09-25 17:03:11,406::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:03:11,414::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00050173 s, 753 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:11,416::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:11,416::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:11,416::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:11,416::__init__::232::IOProcess::(_processLogs) (106825) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:11,417::__init__::232::IOProcess::(_processLogs) (106825) Queuing response >Dummy-148::DEBUG::2014-09-25 17:03:11,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:11,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189739 s, 54.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:11,907::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69456::DEBUG::2014-09-25 17:03:11,908::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:03:11,908::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69456::DEBUG::2014-09-25 17:03:11,910::task::595::Storage.TaskManager.Task::(_updateState) Task=`c9ed5fae-b2df-40a1-871a-d4257aafaa81`::moving from state init -> state preparing >Thread-69456::INFO::2014-09-25 17:03:11,911::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69456::INFO::2014-09-25 17:03:11,911::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69456::DEBUG::2014-09-25 17:03:11,912::task::1191::Storage.TaskManager.Task::(prepare) Task=`c9ed5fae-b2df-40a1-871a-d4257aafaa81`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69456::DEBUG::2014-09-25 17:03:11,912::task::595::Storage.TaskManager.Task::(_updateState) Task=`c9ed5fae-b2df-40a1-871a-d4257aafaa81`::moving from state preparing -> state finished >Thread-69456::DEBUG::2014-09-25 17:03:11,912::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69456::DEBUG::2014-09-25 17:03:11,912::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69456::DEBUG::2014-09-25 17:03:11,912::task::993::Storage.TaskManager.Task::(_decref) Task=`c9ed5fae-b2df-40a1-871a-d4257aafaa81`::ref 0 aborting False >Thread-69456::DEBUG::2014-09-25 17:03:11,913::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69456::DEBUG::2014-09-25 17:03:11,913::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:11,928::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:11,929::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69457::DEBUG::2014-09-25 17:03:11,930::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69457::DEBUG::2014-09-25 17:03:11,931::task::595::Storage.TaskManager.Task::(_updateState) Task=`19a422b0-74b4-440d-84eb-e33db84f793a`::moving from state init -> state preparing >Thread-69457::INFO::2014-09-25 17:03:11,932::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69457::DEBUG::2014-09-25 17:03:11,932::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b9d0b280-8227-4cad-addd-112e6e7578f8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69457::DEBUG::2014-09-25 17:03:11,932::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69457::DEBUG::2014-09-25 17:03:11,933::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69457::DEBUG::2014-09-25 17:03:11,933::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b9d0b280-8227-4cad-addd-112e6e7578f8`::Granted request >Thread-69457::DEBUG::2014-09-25 17:03:11,934::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`19a422b0-74b4-440d-84eb-e33db84f793a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69457::DEBUG::2014-09-25 17:03:11,934::task::993::Storage.TaskManager.Task::(_decref) Task=`19a422b0-74b4-440d-84eb-e33db84f793a`::ref 1 aborting False >Thread-69457::INFO::2014-09-25 17:03:11,938::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69457::DEBUG::2014-09-25 17:03:11,939::task::1191::Storage.TaskManager.Task::(prepare) Task=`19a422b0-74b4-440d-84eb-e33db84f793a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69457::DEBUG::2014-09-25 17:03:11,939::task::595::Storage.TaskManager.Task::(_updateState) Task=`19a422b0-74b4-440d-84eb-e33db84f793a`::moving from state preparing -> state finished >Thread-69457::DEBUG::2014-09-25 17:03:11,939::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69457::DEBUG::2014-09-25 17:03:11,939::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69457::DEBUG::2014-09-25 17:03:11,940::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69457::DEBUG::2014-09-25 17:03:11,940::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69457::DEBUG::2014-09-25 17:03:11,940::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69457::DEBUG::2014-09-25 17:03:11,940::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69457::DEBUG::2014-09-25 17:03:11,941::task::993::Storage.TaskManager.Task::(_decref) Task=`19a422b0-74b4-440d-84eb-e33db84f793a`::ref 0 aborting False >Thread-69457::DEBUG::2014-09-25 17:03:11,941::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75273076736', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69457::DEBUG::2014-09-25 17:03:11,942::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:13,653::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:13,654::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69458::DEBUG::2014-09-25 17:03:13,656::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:13,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:13,827::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0227294 s, 45.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:14,065::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:14,065::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:14,065::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:14,066::__init__::232::IOProcess::(_processLogs) (106826) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:14,070::__init__::232::IOProcess::(_processLogs) (106826) Queuing response >Thread-81::DEBUG::2014-09-25 17:03:14,071::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:03:14,094::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00467233 s, 79.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:14,095::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:14,095::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:14,096::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:14,096::__init__::232::IOProcess::(_processLogs) (106827) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:14,098::__init__::232::IOProcess::(_processLogs) (106827) Queuing response >Dummy-148::DEBUG::2014-09-25 17:03:15,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:15,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164788 s, 62.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:16,674::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:16,678::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69459::DEBUG::2014-09-25 17:03:16,684::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:17,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:17,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0235105 s, 43.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:18,806::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:18,807::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:18,807::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:18,807::__init__::232::IOProcess::(_processLogs) (106828) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:18,808::__init__::232::IOProcess::(_processLogs) (106828) Queuing response >Thread-79::DEBUG::2014-09-25 17:03:18,809::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:03:18,817::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000297168 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:18,819::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:18,819::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:18,819::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:18,819::__init__::232::IOProcess::(_processLogs) (106829) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:18,820::__init__::232::IOProcess::(_processLogs) (106829) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:18,821::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:18,822::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:18,823::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:18,823::__init__::232::IOProcess::(_processLogs) (106830) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:18,823::__init__::232::IOProcess::(_processLogs) (106830) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:18,824::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:18,824::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:18,825::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:18,825::__init__::232::IOProcess::(_processLogs) (106831) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:18,825::__init__::232::IOProcess::(_processLogs) (106831) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:18,826::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:18,826::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:18,827::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:18,827::__init__::232::IOProcess::(_processLogs) (106832) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:18,828::__init__::232::IOProcess::(_processLogs) (106832) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:19,705::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:19,706::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69460::DEBUG::2014-09-25 17:03:19,708::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:19,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:19,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149512 s, 68.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:21,419::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:21,420::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:21,420::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:21,420::__init__::232::IOProcess::(_processLogs) (106833) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:21,421::__init__::232::IOProcess::(_processLogs) (106833) Queuing response >Thread-80::DEBUG::2014-09-25 17:03:21,422::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:03:21,431::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000435297 s, 868 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:21,432::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:21,432::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:21,433::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:21,433::__init__::232::IOProcess::(_processLogs) (106834) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:21,433::__init__::232::IOProcess::(_processLogs) (106834) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:21,986::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:21,987::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69461::DEBUG::2014-09-25 17:03:21,987::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69461::DEBUG::2014-09-25 17:03:21,989::task::595::Storage.TaskManager.Task::(_updateState) Task=`992a11cb-f292-4d3f-820a-7c9975f740e1`::moving from state init -> state preparing >Thread-69461::INFO::2014-09-25 17:03:21,989::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69461::INFO::2014-09-25 17:03:21,990::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69461::DEBUG::2014-09-25 17:03:21,990::task::1191::Storage.TaskManager.Task::(prepare) Task=`992a11cb-f292-4d3f-820a-7c9975f740e1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69461::DEBUG::2014-09-25 17:03:21,990::task::595::Storage.TaskManager.Task::(_updateState) Task=`992a11cb-f292-4d3f-820a-7c9975f740e1`::moving from state preparing -> state finished >Thread-69461::DEBUG::2014-09-25 17:03:21,991::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69461::DEBUG::2014-09-25 17:03:21,991::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69461::DEBUG::2014-09-25 17:03:21,991::task::993::Storage.TaskManager.Task::(_decref) Task=`992a11cb-f292-4d3f-820a-7c9975f740e1`::ref 0 aborting False >Thread-69461::DEBUG::2014-09-25 17:03:21,991::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69461::DEBUG::2014-09-25 17:03:21,992::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:22,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:22,010::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:22,012::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69462::DEBUG::2014-09-25 17:03:22,012::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69462::DEBUG::2014-09-25 17:03:22,014::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d7cebd6-9281-4fc4-b61c-0b200f3bc2fe`::moving from state init -> state preparing >Thread-69462::INFO::2014-09-25 17:03:22,014::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69462::DEBUG::2014-09-25 17:03:22,015::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6763fd85-8045-4a66-a5f6-8fc7e249dc40`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69462::DEBUG::2014-09-25 17:03:22,015::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69462::DEBUG::2014-09-25 17:03:22,015::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69462::DEBUG::2014-09-25 17:03:22,016::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6763fd85-8045-4a66-a5f6-8fc7e249dc40`::Granted request >Thread-69462::DEBUG::2014-09-25 17:03:22,017::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`6d7cebd6-9281-4fc4-b61c-0b200f3bc2fe`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69462::DEBUG::2014-09-25 17:03:22,017::task::993::Storage.TaskManager.Task::(_decref) Task=`6d7cebd6-9281-4fc4-b61c-0b200f3bc2fe`::ref 1 aborting False >Thread-69462::INFO::2014-09-25 17:03:22,025::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75216453632', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75244765184', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69462::DEBUG::2014-09-25 17:03:22,026::task::1191::Storage.TaskManager.Task::(prepare) Task=`6d7cebd6-9281-4fc4-b61c-0b200f3bc2fe`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75216453632', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75244765184', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69462::DEBUG::2014-09-25 17:03:22,027::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d7cebd6-9281-4fc4-b61c-0b200f3bc2fe`::moving from state preparing -> state finished >Thread-69462::DEBUG::2014-09-25 17:03:22,028::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69462::DEBUG::2014-09-25 17:03:22,029::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69462::DEBUG::2014-09-25 17:03:22,031::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69462::DEBUG::2014-09-25 17:03:22,032::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69462::DEBUG::2014-09-25 17:03:22,033::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69462::DEBUG::2014-09-25 17:03:22,034::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69462::DEBUG::2014-09-25 17:03:22,036::task::993::Storage.TaskManager.Task::(_decref) Task=`6d7cebd6-9281-4fc4-b61c-0b200f3bc2fe`::ref 0 aborting False >Thread-69462::DEBUG::2014-09-25 17:03:22,038::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75216453632', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75244765184', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69462::DEBUG::2014-09-25 17:03:22,040::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:22,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0296271 s, 34.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:22,724::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:22,725::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69463::DEBUG::2014-09-25 17:03:22,727::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:24,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:24,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0139254 s, 73.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:24,101::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:24,101::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:24,102::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:24,103::__init__::232::IOProcess::(_processLogs) (106835) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:24,105::__init__::232::IOProcess::(_processLogs) (106835) Queuing response >Thread-81::DEBUG::2014-09-25 17:03:24,106::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:03:24,121::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00251012 s, 148 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:24,124::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:24,125::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:24,125::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:24,126::__init__::232::IOProcess::(_processLogs) (106836) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:24,128::__init__::232::IOProcess::(_processLogs) (106836) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:25,742::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:25,743::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69464::DEBUG::2014-09-25 17:03:25,746::task::595::Storage.TaskManager.Task::(_updateState) Task=`1e1cc878-bbf5-46d2-9898-8207a41d1b65`::moving from state init -> state preparing >Thread-69464::INFO::2014-09-25 17:03:25,746::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69464::INFO::2014-09-25 17:03:25,747::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000297168', 'lastCheck': '6.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000435297', 'lastCheck': '4.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00251012', 'lastCheck': '1.6', 'valid': True}} >Thread-69464::DEBUG::2014-09-25 17:03:25,747::task::1191::Storage.TaskManager.Task::(prepare) Task=`1e1cc878-bbf5-46d2-9898-8207a41d1b65`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000297168', 'lastCheck': '6.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000435297', 'lastCheck': '4.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00251012', 'lastCheck': '1.6', 'valid': True}} >Thread-69464::DEBUG::2014-09-25 17:03:25,747::task::595::Storage.TaskManager.Task::(_updateState) Task=`1e1cc878-bbf5-46d2-9898-8207a41d1b65`::moving from state preparing -> state finished >Thread-69464::DEBUG::2014-09-25 17:03:25,747::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69464::DEBUG::2014-09-25 17:03:25,748::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69464::DEBUG::2014-09-25 17:03:25,748::task::993::Storage.TaskManager.Task::(_decref) Task=`1e1cc878-bbf5-46d2-9898-8207a41d1b65`::ref 0 aborting False >Thread-69464::DEBUG::2014-09-25 17:03:25,750::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:25,773::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:25,774::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69465::DEBUG::2014-09-25 17:03:25,776::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:26,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:26,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0225724 s, 45.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:03:28,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:28,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163099 s, 62.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:28,828::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:28,828::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:28,829::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:28,829::__init__::232::IOProcess::(_processLogs) (106837) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:28,833::__init__::232::IOProcess::(_processLogs) (106837) Queuing response >Thread-79::DEBUG::2014-09-25 17:03:28,834::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:28,842::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:28,844::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69466::DEBUG::2014-09-25 17:03:28,846::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-79::DEBUG::2014-09-25 17:03:28,852::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00896556 s, 62.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:28,853::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:28,854::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:28,854::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:28,854::__init__::232::IOProcess::(_processLogs) (106838) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:28,860::__init__::232::IOProcess::(_processLogs) (106838) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:28,861::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:28,862::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:28,862::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:28,862::__init__::232::IOProcess::(_processLogs) (106839) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:28,863::__init__::232::IOProcess::(_processLogs) (106839) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:28,863::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:28,863::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:28,864::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:28,864::__init__::232::IOProcess::(_processLogs) (106840) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:28,864::__init__::232::IOProcess::(_processLogs) (106840) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:28,865::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:28,865::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:28,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:28,867::__init__::232::IOProcess::(_processLogs) (106841) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:28,867::__init__::232::IOProcess::(_processLogs) (106841) Queuing response >Dummy-148::DEBUG::2014-09-25 17:03:30,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:30,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151182 s, 67.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:31,436::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:31,436::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:31,436::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:31,437::__init__::232::IOProcess::(_processLogs) (106842) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:31,438::__init__::232::IOProcess::(_processLogs) (106842) Queuing response >Thread-80::DEBUG::2014-09-25 17:03:31,439::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:03:31,447::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000462197 s, 818 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:31,449::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:31,449::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:31,449::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:31,449::__init__::232::IOProcess::(_processLogs) (106843) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:31,450::__init__::232::IOProcess::(_processLogs) (106843) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:31,859::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:31,860::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69467::DEBUG::2014-09-25 17:03:31,863::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:32,070::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:32,071::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69468::DEBUG::2014-09-25 17:03:32,071::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69468::DEBUG::2014-09-25 17:03:32,073::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ccf9123-addb-4e68-88e5-f8182f33d867`::moving from state init -> state preparing >Thread-69468::INFO::2014-09-25 17:03:32,073::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69468::INFO::2014-09-25 17:03:32,074::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69468::DEBUG::2014-09-25 17:03:32,074::task::1191::Storage.TaskManager.Task::(prepare) Task=`4ccf9123-addb-4e68-88e5-f8182f33d867`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69468::DEBUG::2014-09-25 17:03:32,074::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ccf9123-addb-4e68-88e5-f8182f33d867`::moving from state preparing -> state finished >Thread-69468::DEBUG::2014-09-25 17:03:32,075::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69468::DEBUG::2014-09-25 17:03:32,075::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69468::DEBUG::2014-09-25 17:03:32,075::task::993::Storage.TaskManager.Task::(_decref) Task=`4ccf9123-addb-4e68-88e5-f8182f33d867`::ref 0 aborting False >Thread-69468::DEBUG::2014-09-25 17:03:32,075::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69468::DEBUG::2014-09-25 17:03:32,076::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:32,086::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69469::DEBUG::2014-09-25 17:03:32,088::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:03:32,088::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69469::DEBUG::2014-09-25 17:03:32,090::task::595::Storage.TaskManager.Task::(_updateState) Task=`ce8f5b4f-7f68-41ee-81b2-22b885e4879a`::moving from state init -> state preparing >Thread-69469::INFO::2014-09-25 17:03:32,090::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69469::DEBUG::2014-09-25 17:03:32,091::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`870894ee-42c3-41df-bce5-38cbecef5ddb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69469::DEBUG::2014-09-25 17:03:32,091::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69469::DEBUG::2014-09-25 17:03:32,091::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69469::DEBUG::2014-09-25 17:03:32,092::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`870894ee-42c3-41df-bce5-38cbecef5ddb`::Granted request >Thread-69469::DEBUG::2014-09-25 17:03:32,092::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ce8f5b4f-7f68-41ee-81b2-22b885e4879a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69469::DEBUG::2014-09-25 17:03:32,092::task::993::Storage.TaskManager.Task::(_decref) Task=`ce8f5b4f-7f68-41ee-81b2-22b885e4879a`::ref 1 aborting False >Thread-69469::INFO::2014-09-25 17:03:32,097::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69469::DEBUG::2014-09-25 17:03:32,097::task::1191::Storage.TaskManager.Task::(prepare) Task=`ce8f5b4f-7f68-41ee-81b2-22b885e4879a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69469::DEBUG::2014-09-25 17:03:32,097::task::595::Storage.TaskManager.Task::(_updateState) Task=`ce8f5b4f-7f68-41ee-81b2-22b885e4879a`::moving from state preparing -> state finished >Thread-69469::DEBUG::2014-09-25 17:03:32,097::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69469::DEBUG::2014-09-25 17:03:32,098::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69469::DEBUG::2014-09-25 17:03:32,098::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69469::DEBUG::2014-09-25 17:03:32,098::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69469::DEBUG::2014-09-25 17:03:32,099::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69469::DEBUG::2014-09-25 17:03:32,099::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69469::DEBUG::2014-09-25 17:03:32,099::task::993::Storage.TaskManager.Task::(_decref) Task=`ce8f5b4f-7f68-41ee-81b2-22b885e4879a`::ref 0 aborting False >Thread-69469::DEBUG::2014-09-25 17:03:32,100::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69469::DEBUG::2014-09-25 17:03:32,100::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:32,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:32,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015515 s, 66.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:34,130::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:34,131::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:34,131::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:34,131::__init__::232::IOProcess::(_processLogs) (106844) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:34,138::__init__::232::IOProcess::(_processLogs) (106844) Queuing response >Thread-81::DEBUG::2014-09-25 17:03:34,139::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:03:34,234::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00958446 s, 38.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:34,235::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:34,236::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:34,236::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:34,236::__init__::232::IOProcess::(_processLogs) (106845) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:34,250::__init__::232::IOProcess::(_processLogs) (106845) Queuing response >Dummy-148::DEBUG::2014-09-25 17:03:34,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:34,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143049 s, 71.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:34,877::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:34,878::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69470::DEBUG::2014-09-25 17:03:34,880::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:36,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:36,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213504 s, 48.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:37,892::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:37,894::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69471::DEBUG::2014-09-25 17:03:37,896::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:38,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:38,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133276 s, 76.8 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 17:03:38,867::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:03:38,871::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,872::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,872::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,872::__init__::232::IOProcess::(_processLogs) (106846) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:03:38,873::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,873::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,873::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,874::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,874::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,874::__init__::232::IOProcess::(_processLogs) (106847) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:03:38,874::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,875::__init__::232::IOProcess::(_processLogs) (106848) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:03:38,875::__init__::232::IOProcess::(_processLogs) (106847) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:38,876::__init__::232::IOProcess::(_processLogs) (106846) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:38,878::__init__::232::IOProcess::(_processLogs) (106848) Queuing response >Thread-79::DEBUG::2014-09-25 17:03:38,882::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:03:38,883::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,883::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,884::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,884::__init__::232::IOProcess::(_processLogs) (106849) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:03:38,915::__init__::232::IOProcess::(_processLogs) (106849) Queuing response >Thread-79::DEBUG::2014-09-25 17:03:38,915::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:03:38,916::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,917::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,917::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,917::__init__::232::IOProcess::(_processLogs) (106850) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:03:38,924::__init__::232::IOProcess::(_processLogs) (106850) Queuing response >Thread-79::DEBUG::2014-09-25 17:03:38,925::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:03:38,926::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,926::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,927::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,927::__init__::232::IOProcess::(_processLogs) (106851) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:38,927::__init__::232::IOProcess::(_processLogs) (106851) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:38,928::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,928::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,928::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,929::__init__::232::IOProcess::(_processLogs) (106852) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:03:38,942::__init__::232::IOProcess::(_processLogs) (106852) Queuing response >Thread-79::DEBUG::2014-09-25 17:03:38,943::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 17:03:38,943::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:03:38,943::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:03:38,944::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,944::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,945::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,945::__init__::232::IOProcess::(_processLogs) (106853) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:38,949::__init__::232::IOProcess::(_processLogs) (106853) Queuing response >Thread-79::DEBUG::2014-09-25 17:03:38,950::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:03:38,974::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00839867 s, 66.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:38,975::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,975::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,976::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,976::__init__::232::IOProcess::(_processLogs) (106854) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:38,981::__init__::232::IOProcess::(_processLogs) (106854) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:38,982::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,982::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,982::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,983::__init__::232::IOProcess::(_processLogs) (106855) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:38,984::__init__::232::IOProcess::(_processLogs) (106855) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:38,985::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,985::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,986::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,986::__init__::232::IOProcess::(_processLogs) (106856) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:38,992::__init__::232::IOProcess::(_processLogs) (106856) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:38,993::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:38,993::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:38,993::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:38,993::__init__::232::IOProcess::(_processLogs) (106857) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:38,995::__init__::232::IOProcess::(_processLogs) (106857) Queuing response >Dummy-148::DEBUG::2014-09-25 17:03:40,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:40,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160886 s, 63.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:40,907::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:40,908::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69475::DEBUG::2014-09-25 17:03:40,911::task::595::Storage.TaskManager.Task::(_updateState) Task=`67b7bfc1-3da1-4f90-a9ba-edcefd49faeb`::moving from state init -> state preparing >Thread-69475::INFO::2014-09-25 17:03:40,911::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69475::INFO::2014-09-25 17:03:40,912::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00839867', 'lastCheck': '1.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000462197', 'lastCheck': '9.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00958446', 'lastCheck': '6.7', 'valid': True}} >Thread-69475::DEBUG::2014-09-25 17:03:40,912::task::1191::Storage.TaskManager.Task::(prepare) Task=`67b7bfc1-3da1-4f90-a9ba-edcefd49faeb`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00839867', 'lastCheck': '1.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000462197', 'lastCheck': '9.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00958446', 'lastCheck': '6.7', 'valid': True}} >Thread-69475::DEBUG::2014-09-25 17:03:40,912::task::595::Storage.TaskManager.Task::(_updateState) Task=`67b7bfc1-3da1-4f90-a9ba-edcefd49faeb`::moving from state preparing -> state finished >Thread-69475::DEBUG::2014-09-25 17:03:40,913::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69475::DEBUG::2014-09-25 17:03:40,913::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69475::DEBUG::2014-09-25 17:03:40,913::task::993::Storage.TaskManager.Task::(_decref) Task=`67b7bfc1-3da1-4f90-a9ba-edcefd49faeb`::ref 0 aborting False >Thread-69475::DEBUG::2014-09-25 17:03:40,915::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:40,930::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:40,932::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69476::DEBUG::2014-09-25 17:03:40,933::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:03:41,452::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:41,452::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:41,453::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:41,453::__init__::232::IOProcess::(_processLogs) (106858) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:41,454::__init__::232::IOProcess::(_processLogs) (106858) Queuing response >Thread-80::DEBUG::2014-09-25 17:03:41,455::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:03:41,464::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000452209 s, 836 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:41,465::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:41,465::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:41,466::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:41,466::__init__::232::IOProcess::(_processLogs) (106859) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:41,466::__init__::232::IOProcess::(_processLogs) (106859) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:42,125::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:42,127::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69477::DEBUG::2014-09-25 17:03:42,127::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69477::DEBUG::2014-09-25 17:03:42,129::task::595::Storage.TaskManager.Task::(_updateState) Task=`effc1580-d3c0-4532-a3c7-955bb20f49f1`::moving from state init -> state preparing >Thread-69477::INFO::2014-09-25 17:03:42,129::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69477::INFO::2014-09-25 17:03:42,129::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69477::DEBUG::2014-09-25 17:03:42,130::task::1191::Storage.TaskManager.Task::(prepare) Task=`effc1580-d3c0-4532-a3c7-955bb20f49f1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69477::DEBUG::2014-09-25 17:03:42,130::task::595::Storage.TaskManager.Task::(_updateState) Task=`effc1580-d3c0-4532-a3c7-955bb20f49f1`::moving from state preparing -> state finished >Thread-69477::DEBUG::2014-09-25 17:03:42,130::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69477::DEBUG::2014-09-25 17:03:42,130::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69477::DEBUG::2014-09-25 17:03:42,130::task::993::Storage.TaskManager.Task::(_decref) Task=`effc1580-d3c0-4532-a3c7-955bb20f49f1`::ref 0 aborting False >Thread-69477::DEBUG::2014-09-25 17:03:42,131::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69477::DEBUG::2014-09-25 17:03:42,131::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:42,149::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69478::DEBUG::2014-09-25 17:03:42,151::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:03:42,150::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69478::DEBUG::2014-09-25 17:03:42,153::task::595::Storage.TaskManager.Task::(_updateState) Task=`3709f142-402e-4a85-99a6-390ba5e06099`::moving from state init -> state preparing >Thread-69478::INFO::2014-09-25 17:03:42,153::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69478::DEBUG::2014-09-25 17:03:42,154::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6f878921-f024-4efa-9607-289cf81d90c4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69478::DEBUG::2014-09-25 17:03:42,154::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69478::DEBUG::2014-09-25 17:03:42,154::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69478::DEBUG::2014-09-25 17:03:42,154::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6f878921-f024-4efa-9607-289cf81d90c4`::Granted request >Thread-69478::DEBUG::2014-09-25 17:03:42,156::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3709f142-402e-4a85-99a6-390ba5e06099`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69478::DEBUG::2014-09-25 17:03:42,156::task::993::Storage.TaskManager.Task::(_decref) Task=`3709f142-402e-4a85-99a6-390ba5e06099`::ref 1 aborting False >Thread-69478::INFO::2014-09-25 17:03:42,161::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69478::DEBUG::2014-09-25 17:03:42,161::task::1191::Storage.TaskManager.Task::(prepare) Task=`3709f142-402e-4a85-99a6-390ba5e06099`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69478::DEBUG::2014-09-25 17:03:42,161::task::595::Storage.TaskManager.Task::(_updateState) Task=`3709f142-402e-4a85-99a6-390ba5e06099`::moving from state preparing -> state finished >Thread-69478::DEBUG::2014-09-25 17:03:42,161::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69478::DEBUG::2014-09-25 17:03:42,162::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69478::DEBUG::2014-09-25 17:03:42,162::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69478::DEBUG::2014-09-25 17:03:42,162::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69478::DEBUG::2014-09-25 17:03:42,163::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69478::DEBUG::2014-09-25 17:03:42,163::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69478::DEBUG::2014-09-25 17:03:42,163::task::993::Storage.TaskManager.Task::(_decref) Task=`3709f142-402e-4a85-99a6-390ba5e06099`::ref 0 aborting False >Thread-69478::DEBUG::2014-09-25 17:03:42,164::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69478::DEBUG::2014-09-25 17:03:42,164::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:42,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:42,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171317 s, 59.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:43,979::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:43,981::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69479::DEBUG::2014-09-25 17:03:43,983::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:03:44,251::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:44,251::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:44,252::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:44,252::__init__::232::IOProcess::(_processLogs) (106860) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:44,253::__init__::232::IOProcess::(_processLogs) (106860) Queuing response >Thread-81::DEBUG::2014-09-25 17:03:44,254::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:03:44,266::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00136543 s, 272 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:44,267::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:44,267::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:44,268::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:44,268::__init__::232::IOProcess::(_processLogs) (106861) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:44,272::__init__::232::IOProcess::(_processLogs) (106861) Queuing response >Dummy-148::DEBUG::2014-09-25 17:03:44,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:44,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133448 s, 76.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:03:46,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:46,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0238056 s, 43.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:46,997::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:47,000::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69480::DEBUG::2014-09-25 17:03:47,005::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:48,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:48,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013147 s, 77.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:48,996::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:48,996::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:48,997::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:48,997::__init__::232::IOProcess::(_processLogs) (106862) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:48,998::__init__::232::IOProcess::(_processLogs) (106862) Queuing response >Thread-79::DEBUG::2014-09-25 17:03:48,999::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:03:49,007::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000410868 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:49,008::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:49,009::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:49,009::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:49,009::__init__::232::IOProcess::(_processLogs) (106863) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:49,010::__init__::232::IOProcess::(_processLogs) (106863) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:49,012::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:49,012::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:49,012::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:49,012::__init__::232::IOProcess::(_processLogs) (106864) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:49,013::__init__::232::IOProcess::(_processLogs) (106864) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:49,013::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:49,014::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:49,014::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:49,014::__init__::232::IOProcess::(_processLogs) (106865) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:49,014::__init__::232::IOProcess::(_processLogs) (106865) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:49,015::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:49,016::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:49,016::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:49,016::__init__::232::IOProcess::(_processLogs) (106866) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:49,017::__init__::232::IOProcess::(_processLogs) (106866) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:50,019::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:50,021::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69481::DEBUG::2014-09-25 17:03:50,023::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:50,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:50,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186169 s, 55.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:51,469::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:51,469::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:51,470::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:51,470::__init__::232::IOProcess::(_processLogs) (106867) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:51,470::__init__::232::IOProcess::(_processLogs) (106867) Queuing response >Thread-80::DEBUG::2014-09-25 17:03:51,472::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:03:51,480::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00049488 s, 764 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:51,481::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:51,482::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:51,482::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:51,482::__init__::232::IOProcess::(_processLogs) (106868) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:51,483::__init__::232::IOProcess::(_processLogs) (106868) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:52,203::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:52,205::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69482::DEBUG::2014-09-25 17:03:52,204::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69482::DEBUG::2014-09-25 17:03:52,207::task::595::Storage.TaskManager.Task::(_updateState) Task=`f8be29ca-ff4f-4edc-83f9-81eedb121d74`::moving from state init -> state preparing >Thread-69482::INFO::2014-09-25 17:03:52,207::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69482::INFO::2014-09-25 17:03:52,207::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69482::DEBUG::2014-09-25 17:03:52,208::task::1191::Storage.TaskManager.Task::(prepare) Task=`f8be29ca-ff4f-4edc-83f9-81eedb121d74`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69482::DEBUG::2014-09-25 17:03:52,208::task::595::Storage.TaskManager.Task::(_updateState) Task=`f8be29ca-ff4f-4edc-83f9-81eedb121d74`::moving from state preparing -> state finished >Thread-69482::DEBUG::2014-09-25 17:03:52,208::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69482::DEBUG::2014-09-25 17:03:52,208::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69482::DEBUG::2014-09-25 17:03:52,209::task::993::Storage.TaskManager.Task::(_decref) Task=`f8be29ca-ff4f-4edc-83f9-81eedb121d74`::ref 0 aborting False >Thread-69482::DEBUG::2014-09-25 17:03:52,209::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69482::DEBUG::2014-09-25 17:03:52,209::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:52,223::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:52,224::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69483::DEBUG::2014-09-25 17:03:52,224::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69483::DEBUG::2014-09-25 17:03:52,226::task::595::Storage.TaskManager.Task::(_updateState) Task=`7a8e7820-8cff-4a26-a0b9-ede3b02483a5`::moving from state init -> state preparing >Thread-69483::INFO::2014-09-25 17:03:52,226::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69483::DEBUG::2014-09-25 17:03:52,227::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6ca273aa-2660-4f90-a7b2-6ae13b42c933`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69483::DEBUG::2014-09-25 17:03:52,227::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69483::DEBUG::2014-09-25 17:03:52,227::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69483::DEBUG::2014-09-25 17:03:52,227::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6ca273aa-2660-4f90-a7b2-6ae13b42c933`::Granted request >Thread-69483::DEBUG::2014-09-25 17:03:52,228::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7a8e7820-8cff-4a26-a0b9-ede3b02483a5`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69483::DEBUG::2014-09-25 17:03:52,228::task::993::Storage.TaskManager.Task::(_decref) Task=`7a8e7820-8cff-4a26-a0b9-ede3b02483a5`::ref 1 aborting False >Thread-69483::INFO::2014-09-25 17:03:52,232::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69483::DEBUG::2014-09-25 17:03:52,233::task::1191::Storage.TaskManager.Task::(prepare) Task=`7a8e7820-8cff-4a26-a0b9-ede3b02483a5`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69483::DEBUG::2014-09-25 17:03:52,233::task::595::Storage.TaskManager.Task::(_updateState) Task=`7a8e7820-8cff-4a26-a0b9-ede3b02483a5`::moving from state preparing -> state finished >Thread-69483::DEBUG::2014-09-25 17:03:52,233::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69483::DEBUG::2014-09-25 17:03:52,234::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69483::DEBUG::2014-09-25 17:03:52,234::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69483::DEBUG::2014-09-25 17:03:52,234::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69483::DEBUG::2014-09-25 17:03:52,234::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69483::DEBUG::2014-09-25 17:03:52,235::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69483::DEBUG::2014-09-25 17:03:52,235::task::993::Storage.TaskManager.Task::(_decref) Task=`7a8e7820-8cff-4a26-a0b9-ede3b02483a5`::ref 0 aborting False >Thread-69483::DEBUG::2014-09-25 17:03:52,235::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69483::DEBUG::2014-09-25 17:03:52,236::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:52,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:52,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134458 s, 76.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:53,037::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:53,038::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69484::DEBUG::2014-09-25 17:03:53,040::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:03:54,274::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:54,274::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:54,274::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:54,275::__init__::232::IOProcess::(_processLogs) (106869) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:54,293::__init__::232::IOProcess::(_processLogs) (106869) Queuing response >Thread-81::DEBUG::2014-09-25 17:03:54,294::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:03:54,322::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00289907 s, 128 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:54,323::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:54,323::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:54,323::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:54,323::__init__::232::IOProcess::(_processLogs) (106870) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:54,328::__init__::232::IOProcess::(_processLogs) (106870) Queuing response >Dummy-148::DEBUG::2014-09-25 17:03:54,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:55,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180511 s, 56.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:56,052::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:56,056::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69485::DEBUG::2014-09-25 17:03:56,064::task::595::Storage.TaskManager.Task::(_updateState) Task=`3fec4f43-8898-4322-8473-a5b7d37a5e96`::moving from state init -> state preparing >Thread-69485::INFO::2014-09-25 17:03:56,066::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69485::INFO::2014-09-25 17:03:56,068::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000410868', 'lastCheck': '7.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00049488', 'lastCheck': '4.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00289907', 'lastCheck': '1.7', 'valid': True}} >Thread-69485::DEBUG::2014-09-25 17:03:56,069::task::1191::Storage.TaskManager.Task::(prepare) Task=`3fec4f43-8898-4322-8473-a5b7d37a5e96`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000410868', 'lastCheck': '7.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00049488', 'lastCheck': '4.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00289907', 'lastCheck': '1.7', 'valid': True}} >Thread-69485::DEBUG::2014-09-25 17:03:56,070::task::595::Storage.TaskManager.Task::(_updateState) Task=`3fec4f43-8898-4322-8473-a5b7d37a5e96`::moving from state preparing -> state finished >Thread-69485::DEBUG::2014-09-25 17:03:56,071::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69485::DEBUG::2014-09-25 17:03:56,072::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69485::DEBUG::2014-09-25 17:03:56,073::task::993::Storage.TaskManager.Task::(_decref) Task=`3fec4f43-8898-4322-8473-a5b7d37a5e96`::ref 0 aborting False >Thread-69485::DEBUG::2014-09-25 17:03:56,083::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:56,101::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:56,103::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69486::DEBUG::2014-09-25 17:03:56,104::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:03:57,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:57,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174271 s, 58.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:59,018::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:59,019::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:59,019::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:59,019::__init__::232::IOProcess::(_processLogs) (106871) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:59,019::__init__::232::IOProcess::(_processLogs) (106871) Queuing response >Thread-79::DEBUG::2014-09-25 17:03:59,021::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:03:59,030::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00045061 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:03:59,031::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:59,031::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:59,031::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:59,032::__init__::232::IOProcess::(_processLogs) (106872) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:03:59,032::__init__::232::IOProcess::(_processLogs) (106872) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:59,034::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:59,034::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:59,035::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:59,035::__init__::232::IOProcess::(_processLogs) (106873) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:59,035::__init__::232::IOProcess::(_processLogs) (106873) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:59,036::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:59,036::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:59,037::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:59,037::__init__::232::IOProcess::(_processLogs) (106874) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:59,038::__init__::232::IOProcess::(_processLogs) (106874) Queuing response >Thread-29::DEBUG::2014-09-25 17:03:59,038::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:03:59,038::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:03:59,039::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:03:59,040::__init__::232::IOProcess::(_processLogs) (106875) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:03:59,040::__init__::232::IOProcess::(_processLogs) (106875) Queuing response >Dummy-148::DEBUG::2014-09-25 17:03:59,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:03:59,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0233954 s, 43.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:03:59,146::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:03:59,147::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69487::DEBUG::2014-09-25 17:03:59,149::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:01,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:01,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0208843 s, 49.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:01,485::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:01,485::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:01,486::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:01,486::__init__::232::IOProcess::(_processLogs) (106876) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:01,486::__init__::232::IOProcess::(_processLogs) (106876) Queuing response >Thread-80::DEBUG::2014-09-25 17:04:01,488::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:04:01,497::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000528086 s, 716 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:01,498::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:01,498::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:01,499::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:01,501::__init__::232::IOProcess::(_processLogs) (106877) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:01,501::__init__::232::IOProcess::(_processLogs) (106877) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:02,166::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:02,168::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69488::DEBUG::2014-09-25 17:04:02,169::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:02,270::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:02,272::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69489::DEBUG::2014-09-25 17:04:02,272::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69489::DEBUG::2014-09-25 17:04:02,273::task::595::Storage.TaskManager.Task::(_updateState) Task=`57c74a49-dfe8-4e1e-9868-a47f05b2e650`::moving from state init -> state preparing >Thread-69489::INFO::2014-09-25 17:04:02,274::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69489::INFO::2014-09-25 17:04:02,274::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69489::DEBUG::2014-09-25 17:04:02,274::task::1191::Storage.TaskManager.Task::(prepare) Task=`57c74a49-dfe8-4e1e-9868-a47f05b2e650`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69489::DEBUG::2014-09-25 17:04:02,275::task::595::Storage.TaskManager.Task::(_updateState) Task=`57c74a49-dfe8-4e1e-9868-a47f05b2e650`::moving from state preparing -> state finished >Thread-69489::DEBUG::2014-09-25 17:04:02,275::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69489::DEBUG::2014-09-25 17:04:02,275::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69489::DEBUG::2014-09-25 17:04:02,275::task::993::Storage.TaskManager.Task::(_decref) Task=`57c74a49-dfe8-4e1e-9868-a47f05b2e650`::ref 0 aborting False >Thread-69489::DEBUG::2014-09-25 17:04:02,276::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69489::DEBUG::2014-09-25 17:04:02,276::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:02,297::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:02,299::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69490::DEBUG::2014-09-25 17:04:02,299::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69490::DEBUG::2014-09-25 17:04:02,300::task::595::Storage.TaskManager.Task::(_updateState) Task=`d8e5ed80-b12d-4b24-944d-f2e529a3888a`::moving from state init -> state preparing >Thread-69490::INFO::2014-09-25 17:04:02,301::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69490::DEBUG::2014-09-25 17:04:02,301::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c64a2d8a-f268-4dfb-aca0-8b1fa60c1861`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69490::DEBUG::2014-09-25 17:04:02,302::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69490::DEBUG::2014-09-25 17:04:02,302::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69490::DEBUG::2014-09-25 17:04:02,302::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c64a2d8a-f268-4dfb-aca0-8b1fa60c1861`::Granted request >Thread-69490::DEBUG::2014-09-25 17:04:02,303::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`d8e5ed80-b12d-4b24-944d-f2e529a3888a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69490::DEBUG::2014-09-25 17:04:02,303::task::993::Storage.TaskManager.Task::(_decref) Task=`d8e5ed80-b12d-4b24-944d-f2e529a3888a`::ref 1 aborting False >Thread-69490::INFO::2014-09-25 17:04:02,307::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69490::DEBUG::2014-09-25 17:04:02,308::task::1191::Storage.TaskManager.Task::(prepare) Task=`d8e5ed80-b12d-4b24-944d-f2e529a3888a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69490::DEBUG::2014-09-25 17:04:02,308::task::595::Storage.TaskManager.Task::(_updateState) Task=`d8e5ed80-b12d-4b24-944d-f2e529a3888a`::moving from state preparing -> state finished >Thread-69490::DEBUG::2014-09-25 17:04:02,308::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69490::DEBUG::2014-09-25 17:04:02,308::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69490::DEBUG::2014-09-25 17:04:02,309::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69490::DEBUG::2014-09-25 17:04:02,309::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69490::DEBUG::2014-09-25 17:04:02,309::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69490::DEBUG::2014-09-25 17:04:02,309::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69490::DEBUG::2014-09-25 17:04:02,310::task::993::Storage.TaskManager.Task::(_decref) Task=`d8e5ed80-b12d-4b24-944d-f2e529a3888a`::ref 0 aborting False >Thread-69490::DEBUG::2014-09-25 17:04:02,310::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69490::DEBUG::2014-09-25 17:04:02,311::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:03,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:03,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188574 s, 54.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:04,330::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:04,330::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:04,330::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:04,331::__init__::232::IOProcess::(_processLogs) (106878) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:04,336::__init__::232::IOProcess::(_processLogs) (106878) Queuing response >Thread-81::DEBUG::2014-09-25 17:04:04,337::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:04:04,361::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00719403 s, 51.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:04,362::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:04,362::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:04,362::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:04,363::__init__::232::IOProcess::(_processLogs) (106879) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:04,371::__init__::232::IOProcess::(_processLogs) (106879) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:05,185::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:05,187::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69491::DEBUG::2014-09-25 17:04:05,189::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:05,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:05,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161664 s, 63.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:04:07,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:07,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148116 s, 69.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:08,203::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:08,205::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69492::DEBUG::2014-09-25 17:04:08,207::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:04:09,040::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:09,041::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:09,041::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:09,041::__init__::232::IOProcess::(_processLogs) (106880) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:09,051::__init__::232::IOProcess::(_processLogs) (106880) Queuing response >Thread-79::DEBUG::2014-09-25 17:04:09,053::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:04:09,073::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00442352 s, 127 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:09,074::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:09,075::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:09,075::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:09,075::__init__::232::IOProcess::(_processLogs) (106881) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:09,081::__init__::232::IOProcess::(_processLogs) (106881) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:09,083::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:09,083::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:09,083::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:09,084::__init__::232::IOProcess::(_processLogs) (106882) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:09,084::__init__::232::IOProcess::(_processLogs) (106882) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:09,085::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:09,085::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:09,085::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:09,085::__init__::232::IOProcess::(_processLogs) (106883) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:09,086::__init__::232::IOProcess::(_processLogs) (106883) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:09,087::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:09,087::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:09,088::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:09,088::__init__::232::IOProcess::(_processLogs) (106884) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:09,089::__init__::232::IOProcess::(_processLogs) (106884) Queuing response >Dummy-148::DEBUG::2014-09-25 17:04:09,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:09,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170704 s, 60.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:11,220::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:11,222::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69493::DEBUG::2014-09-25 17:04:11,224::task::595::Storage.TaskManager.Task::(_updateState) Task=`43d66328-f652-4acf-8551-0bd214d91ba8`::moving from state init -> state preparing >Thread-69493::INFO::2014-09-25 17:04:11,224::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69493::INFO::2014-09-25 17:04:11,225::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00442352', 'lastCheck': '2.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000528086', 'lastCheck': '9.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00719403', 'lastCheck': '6.9', 'valid': True}} >Thread-69493::DEBUG::2014-09-25 17:04:11,225::task::1191::Storage.TaskManager.Task::(prepare) Task=`43d66328-f652-4acf-8551-0bd214d91ba8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00442352', 'lastCheck': '2.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000528086', 'lastCheck': '9.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00719403', 'lastCheck': '6.9', 'valid': True}} >Thread-69493::DEBUG::2014-09-25 17:04:11,225::task::595::Storage.TaskManager.Task::(_updateState) Task=`43d66328-f652-4acf-8551-0bd214d91ba8`::moving from state preparing -> state finished >Thread-69493::DEBUG::2014-09-25 17:04:11,226::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69493::DEBUG::2014-09-25 17:04:11,226::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69493::DEBUG::2014-09-25 17:04:11,226::task::993::Storage.TaskManager.Task::(_decref) Task=`43d66328-f652-4acf-8551-0bd214d91ba8`::ref 0 aborting False >Thread-69493::DEBUG::2014-09-25 17:04:11,228::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:11,242::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:11,243::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69494::DEBUG::2014-09-25 17:04:11,245::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-80::DEBUG::2014-09-25 17:04:11,501::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:04:11,504::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:11,505::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:11,505::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:11,506::__init__::232::IOProcess::(_processLogs) (106885) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:04:11,506::__init__::232::IOProcess::(_processLogs) (106885) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:11,507::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:11,507::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:11,508::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:11,508::__init__::232::IOProcess::(_processLogs) (106886) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:04:11,509::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:11,510::__init__::232::IOProcess::(_processLogs) (106886) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:11,511::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:11,513::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:11,514::__init__::232::IOProcess::(_processLogs) (106887) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:04:11,515::__init__::232::IOProcess::(_processLogs) (106887) Queuing response >Thread-80::DEBUG::2014-09-25 17:04:11,519::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:04:11,520::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:11,520::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:11,520::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:11,521::__init__::232::IOProcess::(_processLogs) (106888) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:04:11,524::__init__::232::IOProcess::(_processLogs) (106888) Queuing response >Thread-80::DEBUG::2014-09-25 17:04:11,524::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:04:11,525::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:11,525::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:11,525::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:11,526::__init__::232::IOProcess::(_processLogs) (106889) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:04:11,526::__init__::232::IOProcess::(_processLogs) (106889) Queuing response >Thread-80::DEBUG::2014-09-25 17:04:11,526::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:04:11,528::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:11,528::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:11,529::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:11,529::__init__::232::IOProcess::(_processLogs) (106890) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:11,530::__init__::232::IOProcess::(_processLogs) (106890) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:11,530::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:11,530::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:11,531::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:11,531::__init__::232::IOProcess::(_processLogs) (106891) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:04:11,531::__init__::232::IOProcess::(_processLogs) (106891) Queuing response >Thread-80::DEBUG::2014-09-25 17:04:11,531::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-80::INFO::2014-09-25 17:04:11,533::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:04:11,533::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:04:11,533::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:11,534::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:11,534::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:11,534::__init__::232::IOProcess::(_processLogs) (106892) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:11,535::__init__::232::IOProcess::(_processLogs) (106892) Queuing response >Thread-80::DEBUG::2014-09-25 17:04:11,536::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:04:11,544::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000364842 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:11,546::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:11,546::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:11,547::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:11,548::__init__::232::IOProcess::(_processLogs) (106893) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:11,548::__init__::232::IOProcess::(_processLogs) (106893) Queuing response >Dummy-148::DEBUG::2014-09-25 17:04:11,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:11,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135299 s, 75.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:12,342::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:12,344::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69498::DEBUG::2014-09-25 17:04:12,344::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69498::DEBUG::2014-09-25 17:04:12,346::task::595::Storage.TaskManager.Task::(_updateState) Task=`0704b84a-2554-440c-ac1b-13967cb0e9fa`::moving from state init -> state preparing >Thread-69498::INFO::2014-09-25 17:04:12,347::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69498::INFO::2014-09-25 17:04:12,347::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69498::DEBUG::2014-09-25 17:04:12,347::task::1191::Storage.TaskManager.Task::(prepare) Task=`0704b84a-2554-440c-ac1b-13967cb0e9fa`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69498::DEBUG::2014-09-25 17:04:12,348::task::595::Storage.TaskManager.Task::(_updateState) Task=`0704b84a-2554-440c-ac1b-13967cb0e9fa`::moving from state preparing -> state finished >Thread-69498::DEBUG::2014-09-25 17:04:12,348::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69498::DEBUG::2014-09-25 17:04:12,348::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69498::DEBUG::2014-09-25 17:04:12,348::task::993::Storage.TaskManager.Task::(_decref) Task=`0704b84a-2554-440c-ac1b-13967cb0e9fa`::ref 0 aborting False >Thread-69498::DEBUG::2014-09-25 17:04:12,349::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69498::DEBUG::2014-09-25 17:04:12,349::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:12,363::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:12,364::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69499::DEBUG::2014-09-25 17:04:12,364::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69499::DEBUG::2014-09-25 17:04:12,366::task::595::Storage.TaskManager.Task::(_updateState) Task=`38fccbd7-4c61-40c3-b9df-f27a265aada1`::moving from state init -> state preparing >Thread-69499::INFO::2014-09-25 17:04:12,366::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69499::DEBUG::2014-09-25 17:04:12,367::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f98e554b-cded-4c30-a36a-280e2519f003`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69499::DEBUG::2014-09-25 17:04:12,367::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69499::DEBUG::2014-09-25 17:04:12,368::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69499::DEBUG::2014-09-25 17:04:12,368::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f98e554b-cded-4c30-a36a-280e2519f003`::Granted request >Thread-69499::DEBUG::2014-09-25 17:04:12,369::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`38fccbd7-4c61-40c3-b9df-f27a265aada1`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69499::DEBUG::2014-09-25 17:04:12,369::task::993::Storage.TaskManager.Task::(_decref) Task=`38fccbd7-4c61-40c3-b9df-f27a265aada1`::ref 1 aborting False >Thread-69499::INFO::2014-09-25 17:04:12,373::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69499::DEBUG::2014-09-25 17:04:12,373::task::1191::Storage.TaskManager.Task::(prepare) Task=`38fccbd7-4c61-40c3-b9df-f27a265aada1`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69499::DEBUG::2014-09-25 17:04:12,374::task::595::Storage.TaskManager.Task::(_updateState) Task=`38fccbd7-4c61-40c3-b9df-f27a265aada1`::moving from state preparing -> state finished >Thread-69499::DEBUG::2014-09-25 17:04:12,374::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69499::DEBUG::2014-09-25 17:04:12,374::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69499::DEBUG::2014-09-25 17:04:12,375::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69499::DEBUG::2014-09-25 17:04:12,375::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69499::DEBUG::2014-09-25 17:04:12,375::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69499::DEBUG::2014-09-25 17:04:12,375::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69499::DEBUG::2014-09-25 17:04:12,376::task::993::Storage.TaskManager.Task::(_decref) Task=`38fccbd7-4c61-40c3-b9df-f27a265aada1`::ref 0 aborting False >Thread-69499::DEBUG::2014-09-25 17:04:12,376::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69499::DEBUG::2014-09-25 17:04:12,377::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:13,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:13,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158891 s, 64.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:14,298::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:14,299::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69500::DEBUG::2014-09-25 17:04:14,301::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:04:14,372::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:14,373::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:14,373::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:14,373::__init__::232::IOProcess::(_processLogs) (106894) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:14,385::__init__::232::IOProcess::(_processLogs) (106894) Queuing response >Thread-81::DEBUG::2014-09-25 17:04:14,386::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:04:14,407::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00540022 s, 68.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:14,408::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:14,408::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:14,409::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:14,409::__init__::232::IOProcess::(_processLogs) (106895) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:14,410::__init__::232::IOProcess::(_processLogs) (106895) Queuing response >Dummy-148::DEBUG::2014-09-25 17:04:15,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:15,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160422 s, 63.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:17,319::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:17,320::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69501::DEBUG::2014-09-25 17:04:17,321::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:17,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:17,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182057 s, 56.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:19,089::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:19,089::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:19,089::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:19,090::__init__::232::IOProcess::(_processLogs) (106896) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:19,091::__init__::232::IOProcess::(_processLogs) (106896) Queuing response >Thread-79::DEBUG::2014-09-25 17:04:19,093::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:04:19,108::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00270627 s, 208 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:19,109::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:19,109::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:19,109::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:19,110::__init__::232::IOProcess::(_processLogs) (106897) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:19,113::__init__::232::IOProcess::(_processLogs) (106897) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:19,114::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:19,115::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:19,116::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:19,116::__init__::232::IOProcess::(_processLogs) (106898) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:19,116::__init__::232::IOProcess::(_processLogs) (106898) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:19,116::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:19,117::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:19,118::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:19,118::__init__::232::IOProcess::(_processLogs) (106899) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:19,118::__init__::232::IOProcess::(_processLogs) (106899) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:19,119::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:19,119::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:19,120::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:19,120::__init__::232::IOProcess::(_processLogs) (106900) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:19,120::__init__::232::IOProcess::(_processLogs) (106900) Queuing response >Dummy-148::DEBUG::2014-09-25 17:04:19,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:19,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161753 s, 63.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:20,336::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:20,338::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69502::DEBUG::2014-09-25 17:04:20,340::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:04:21,549::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:21,549::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:21,549::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:21,550::__init__::232::IOProcess::(_processLogs) (106901) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:21,550::__init__::232::IOProcess::(_processLogs) (106901) Queuing response >Thread-80::DEBUG::2014-09-25 17:04:21,551::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:04:21,560::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000471853 s, 801 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:21,561::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:21,562::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:21,562::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:21,562::__init__::232::IOProcess::(_processLogs) (106902) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:21,563::__init__::232::IOProcess::(_processLogs) (106902) Queuing response >Dummy-148::DEBUG::2014-09-25 17:04:21,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:21,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218575 s, 46.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:22,408::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69503::DEBUG::2014-09-25 17:04:22,409::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:04:22,409::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69503::DEBUG::2014-09-25 17:04:22,413::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2a1bcbd-8073-49ec-8c41-da53251d2856`::moving from state init -> state preparing >Thread-69503::INFO::2014-09-25 17:04:22,413::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69503::INFO::2014-09-25 17:04:22,414::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69503::DEBUG::2014-09-25 17:04:22,414::task::1191::Storage.TaskManager.Task::(prepare) Task=`c2a1bcbd-8073-49ec-8c41-da53251d2856`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69503::DEBUG::2014-09-25 17:04:22,414::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2a1bcbd-8073-49ec-8c41-da53251d2856`::moving from state preparing -> state finished >Thread-69503::DEBUG::2014-09-25 17:04:22,414::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69503::DEBUG::2014-09-25 17:04:22,415::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69503::DEBUG::2014-09-25 17:04:22,415::task::993::Storage.TaskManager.Task::(_decref) Task=`c2a1bcbd-8073-49ec-8c41-da53251d2856`::ref 0 aborting False >Thread-69503::DEBUG::2014-09-25 17:04:22,415::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69503::DEBUG::2014-09-25 17:04:22,416::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:22,430::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69504::DEBUG::2014-09-25 17:04:22,431::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:04:22,431::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69504::DEBUG::2014-09-25 17:04:22,433::task::595::Storage.TaskManager.Task::(_updateState) Task=`6aa707d2-b197-416c-9796-9db11692c2c0`::moving from state init -> state preparing >Thread-69504::INFO::2014-09-25 17:04:22,434::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69504::DEBUG::2014-09-25 17:04:22,434::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`830d1cfd-4b12-4436-a556-ba50db3940c2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69504::DEBUG::2014-09-25 17:04:22,435::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69504::DEBUG::2014-09-25 17:04:22,435::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69504::DEBUG::2014-09-25 17:04:22,435::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`830d1cfd-4b12-4436-a556-ba50db3940c2`::Granted request >Thread-69504::DEBUG::2014-09-25 17:04:22,436::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`6aa707d2-b197-416c-9796-9db11692c2c0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69504::DEBUG::2014-09-25 17:04:22,436::task::993::Storage.TaskManager.Task::(_decref) Task=`6aa707d2-b197-416c-9796-9db11692c2c0`::ref 1 aborting False >Thread-69504::INFO::2014-09-25 17:04:22,440::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69504::DEBUG::2014-09-25 17:04:22,441::task::1191::Storage.TaskManager.Task::(prepare) Task=`6aa707d2-b197-416c-9796-9db11692c2c0`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69504::DEBUG::2014-09-25 17:04:22,441::task::595::Storage.TaskManager.Task::(_updateState) Task=`6aa707d2-b197-416c-9796-9db11692c2c0`::moving from state preparing -> state finished >Thread-69504::DEBUG::2014-09-25 17:04:22,441::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69504::DEBUG::2014-09-25 17:04:22,441::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69504::DEBUG::2014-09-25 17:04:22,442::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69504::DEBUG::2014-09-25 17:04:22,442::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69504::DEBUG::2014-09-25 17:04:22,442::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69504::DEBUG::2014-09-25 17:04:22,442::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69504::DEBUG::2014-09-25 17:04:22,443::task::993::Storage.TaskManager.Task::(_decref) Task=`6aa707d2-b197-416c-9796-9db11692c2c0`::ref 0 aborting False >Thread-69504::DEBUG::2014-09-25 17:04:22,443::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69504::DEBUG::2014-09-25 17:04:22,444::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:23,356::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:23,358::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69505::DEBUG::2014-09-25 17:04:23,359::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:23,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:23,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017503 s, 58.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:24,412::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:24,413::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:24,413::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:24,414::__init__::232::IOProcess::(_processLogs) (106903) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:24,415::__init__::232::IOProcess::(_processLogs) (106903) Queuing response >Thread-81::DEBUG::2014-09-25 17:04:24,416::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:04:24,438::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00686403 s, 54.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:24,439::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:24,439::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:24,440::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:24,440::__init__::232::IOProcess::(_processLogs) (106904) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:24,451::__init__::232::IOProcess::(_processLogs) (106904) Queuing response >Dummy-148::DEBUG::2014-09-25 17:04:25,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:25,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165288 s, 62.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:26,373::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:26,374::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69506::DEBUG::2014-09-25 17:04:26,377::task::595::Storage.TaskManager.Task::(_updateState) Task=`d3d081be-cef0-4e5b-a6d3-c4ab7224a694`::moving from state init -> state preparing >Thread-69506::INFO::2014-09-25 17:04:26,378::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69506::INFO::2014-09-25 17:04:26,378::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00270627', 'lastCheck': '7.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000471853', 'lastCheck': '4.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00686403', 'lastCheck': '1.9', 'valid': True}} >Thread-69506::DEBUG::2014-09-25 17:04:26,378::task::1191::Storage.TaskManager.Task::(prepare) Task=`d3d081be-cef0-4e5b-a6d3-c4ab7224a694`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00270627', 'lastCheck': '7.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000471853', 'lastCheck': '4.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00686403', 'lastCheck': '1.9', 'valid': True}} >Thread-69506::DEBUG::2014-09-25 17:04:26,379::task::595::Storage.TaskManager.Task::(_updateState) Task=`d3d081be-cef0-4e5b-a6d3-c4ab7224a694`::moving from state preparing -> state finished >Thread-69506::DEBUG::2014-09-25 17:04:26,379::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69506::DEBUG::2014-09-25 17:04:26,379::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69506::DEBUG::2014-09-25 17:04:26,379::task::993::Storage.TaskManager.Task::(_decref) Task=`d3d081be-cef0-4e5b-a6d3-c4ab7224a694`::ref 0 aborting False >Thread-69506::DEBUG::2014-09-25 17:04:26,381::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:26,395::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:26,397::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69507::DEBUG::2014-09-25 17:04:26,399::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:27,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:28,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154751 s, 66.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:29,121::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:29,121::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:29,121::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:29,122::__init__::232::IOProcess::(_processLogs) (106905) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:29,127::__init__::232::IOProcess::(_processLogs) (106905) Queuing response >Thread-79::DEBUG::2014-09-25 17:04:29,128::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:04:29,148::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00446501 s, 126 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:29,149::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:29,150::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:29,150::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:29,150::__init__::232::IOProcess::(_processLogs) (106906) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:29,153::__init__::232::IOProcess::(_processLogs) (106906) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:29,154::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:29,154::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:29,155::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:29,156::__init__::232::IOProcess::(_processLogs) (106907) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:29,156::__init__::232::IOProcess::(_processLogs) (106907) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:29,156::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:29,157::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:29,157::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:29,158::__init__::232::IOProcess::(_processLogs) (106908) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:29,158::__init__::232::IOProcess::(_processLogs) (106908) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:29,158::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:29,158::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:29,159::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:29,159::__init__::232::IOProcess::(_processLogs) (106909) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:29,160::__init__::232::IOProcess::(_processLogs) (106909) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:29,449::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:29,450::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69508::DEBUG::2014-09-25 17:04:29,452::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:30,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:30,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149632 s, 68.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:31,565::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:31,566::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:31,566::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:31,566::__init__::232::IOProcess::(_processLogs) (106910) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:31,566::__init__::232::IOProcess::(_processLogs) (106910) Queuing response >Thread-80::DEBUG::2014-09-25 17:04:31,568::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:04:31,577::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000421773 s, 896 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:31,578::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:31,578::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:31,579::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:31,579::__init__::232::IOProcess::(_processLogs) (106911) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:31,580::__init__::232::IOProcess::(_processLogs) (106911) Queuing response >Dummy-148::DEBUG::2014-09-25 17:04:32,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:32,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203803 s, 50.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:32,468::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:32,469::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69509::DEBUG::2014-09-25 17:04:32,471::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:32,473::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:32,474::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69510::DEBUG::2014-09-25 17:04:32,474::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69510::DEBUG::2014-09-25 17:04:32,476::task::595::Storage.TaskManager.Task::(_updateState) Task=`45fb06a8-1613-47b8-bcea-4299b7d0d8af`::moving from state init -> state preparing >Thread-69510::INFO::2014-09-25 17:04:32,476::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69510::INFO::2014-09-25 17:04:32,477::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69510::DEBUG::2014-09-25 17:04:32,477::task::1191::Storage.TaskManager.Task::(prepare) Task=`45fb06a8-1613-47b8-bcea-4299b7d0d8af`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69510::DEBUG::2014-09-25 17:04:32,477::task::595::Storage.TaskManager.Task::(_updateState) Task=`45fb06a8-1613-47b8-bcea-4299b7d0d8af`::moving from state preparing -> state finished >Thread-69510::DEBUG::2014-09-25 17:04:32,477::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69510::DEBUG::2014-09-25 17:04:32,478::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69510::DEBUG::2014-09-25 17:04:32,478::task::993::Storage.TaskManager.Task::(_decref) Task=`45fb06a8-1613-47b8-bcea-4299b7d0d8af`::ref 0 aborting False >Thread-69510::DEBUG::2014-09-25 17:04:32,478::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69510::DEBUG::2014-09-25 17:04:32,478::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:32,491::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:32,493::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69511::DEBUG::2014-09-25 17:04:32,493::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69511::DEBUG::2014-09-25 17:04:32,494::task::595::Storage.TaskManager.Task::(_updateState) Task=`70b9de1f-9f0c-42c2-bf72-4d6dc866bad4`::moving from state init -> state preparing >Thread-69511::INFO::2014-09-25 17:04:32,495::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69511::DEBUG::2014-09-25 17:04:32,495::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3ffa1e7e-7062-4fc9-8699-89923f89d009`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69511::DEBUG::2014-09-25 17:04:32,496::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69511::DEBUG::2014-09-25 17:04:32,496::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69511::DEBUG::2014-09-25 17:04:32,496::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3ffa1e7e-7062-4fc9-8699-89923f89d009`::Granted request >Thread-69511::DEBUG::2014-09-25 17:04:32,497::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`70b9de1f-9f0c-42c2-bf72-4d6dc866bad4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69511::DEBUG::2014-09-25 17:04:32,497::task::993::Storage.TaskManager.Task::(_decref) Task=`70b9de1f-9f0c-42c2-bf72-4d6dc866bad4`::ref 1 aborting False >Thread-69511::INFO::2014-09-25 17:04:32,501::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69511::DEBUG::2014-09-25 17:04:32,502::task::1191::Storage.TaskManager.Task::(prepare) Task=`70b9de1f-9f0c-42c2-bf72-4d6dc866bad4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69511::DEBUG::2014-09-25 17:04:32,502::task::595::Storage.TaskManager.Task::(_updateState) Task=`70b9de1f-9f0c-42c2-bf72-4d6dc866bad4`::moving from state preparing -> state finished >Thread-69511::DEBUG::2014-09-25 17:04:32,502::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69511::DEBUG::2014-09-25 17:04:32,502::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69511::DEBUG::2014-09-25 17:04:32,503::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69511::DEBUG::2014-09-25 17:04:32,503::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69511::DEBUG::2014-09-25 17:04:32,503::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69511::DEBUG::2014-09-25 17:04:32,503::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69511::DEBUG::2014-09-25 17:04:32,504::task::993::Storage.TaskManager.Task::(_decref) Task=`70b9de1f-9f0c-42c2-bf72-4d6dc866bad4`::ref 0 aborting False >Thread-69511::DEBUG::2014-09-25 17:04:32,504::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69511::DEBUG::2014-09-25 17:04:32,505::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:34,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:34,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154606 s, 66.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:34,454::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:34,454::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:34,454::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:34,454::__init__::232::IOProcess::(_processLogs) (106912) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:34,457::__init__::232::IOProcess::(_processLogs) (106912) Queuing response >Thread-81::DEBUG::2014-09-25 17:04:34,458::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:04:34,473::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00102767 s, 361 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:34,474::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:34,475::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:34,475::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:34,475::__init__::232::IOProcess::(_processLogs) (106913) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:34,478::__init__::232::IOProcess::(_processLogs) (106913) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:35,485::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:35,487::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69512::DEBUG::2014-09-25 17:04:35,489::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:36,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:36,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160459 s, 63.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:04:38,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:38,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019458 s, 52.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:38,505::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:38,507::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69513::DEBUG::2014-09-25 17:04:38,509::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:04:39,161::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:39,161::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:39,162::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:39,162::__init__::232::IOProcess::(_processLogs) (106914) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:39,168::__init__::232::IOProcess::(_processLogs) (106914) Queuing response >Thread-79::DEBUG::2014-09-25 17:04:39,169::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:04:39,195::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00996923 s, 56.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:39,196::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:39,196::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:39,196::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:39,197::__init__::232::IOProcess::(_processLogs) (106915) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:39,200::__init__::232::IOProcess::(_processLogs) (106915) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:39,201::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:39,202::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:39,202::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:39,202::__init__::232::IOProcess::(_processLogs) (106916) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:39,205::__init__::232::IOProcess::(_processLogs) (106916) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:39,206::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:39,206::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:39,206::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:39,207::__init__::232::IOProcess::(_processLogs) (106917) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:39,208::__init__::232::IOProcess::(_processLogs) (106917) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:39,209::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:39,209::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:39,209::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:39,209::__init__::232::IOProcess::(_processLogs) (106918) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:39,210::__init__::232::IOProcess::(_processLogs) (106918) Queuing response >Dummy-148::DEBUG::2014-09-25 17:04:40,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:40,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191949 s, 53.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:41,524::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:41,526::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69514::DEBUG::2014-09-25 17:04:41,528::task::595::Storage.TaskManager.Task::(_updateState) Task=`6843b1fd-a7f0-41d5-9642-d2cdecb4e756`::moving from state init -> state preparing >Thread-69514::INFO::2014-09-25 17:04:41,529::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69514::INFO::2014-09-25 17:04:41,529::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00996923', 'lastCheck': '2.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000421773', 'lastCheck': '9.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00102767', 'lastCheck': '7.1', 'valid': True}} >Thread-69514::DEBUG::2014-09-25 17:04:41,530::task::1191::Storage.TaskManager.Task::(prepare) Task=`6843b1fd-a7f0-41d5-9642-d2cdecb4e756`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00996923', 'lastCheck': '2.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000421773', 'lastCheck': '9.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00102767', 'lastCheck': '7.1', 'valid': True}} >Thread-69514::DEBUG::2014-09-25 17:04:41,530::task::595::Storage.TaskManager.Task::(_updateState) Task=`6843b1fd-a7f0-41d5-9642-d2cdecb4e756`::moving from state preparing -> state finished >Thread-69514::DEBUG::2014-09-25 17:04:41,530::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69514::DEBUG::2014-09-25 17:04:41,530::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69514::DEBUG::2014-09-25 17:04:41,530::task::993::Storage.TaskManager.Task::(_decref) Task=`6843b1fd-a7f0-41d5-9642-d2cdecb4e756`::ref 0 aborting False >Thread-69514::DEBUG::2014-09-25 17:04:41,533::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:41,551::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:41,552::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69515::DEBUG::2014-09-25 17:04:41,554::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:04:41,581::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:41,582::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:41,582::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:41,582::__init__::232::IOProcess::(_processLogs) (106919) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:41,583::__init__::232::IOProcess::(_processLogs) (106919) Queuing response >Thread-80::DEBUG::2014-09-25 17:04:41,584::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:04:41,593::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000391657 s, 965 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:41,594::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:41,595::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:41,595::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:41,595::__init__::232::IOProcess::(_processLogs) (106920) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:41,596::__init__::232::IOProcess::(_processLogs) (106920) Queuing response >Dummy-148::DEBUG::2014-09-25 17:04:42,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:42,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013535 s, 75.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:42,571::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:42,573::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69516::DEBUG::2014-09-25 17:04:42,573::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69516::DEBUG::2014-09-25 17:04:42,575::task::595::Storage.TaskManager.Task::(_updateState) Task=`38f23a13-6bcf-473e-9471-d4978ebc775b`::moving from state init -> state preparing >Thread-69516::INFO::2014-09-25 17:04:42,575::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69516::INFO::2014-09-25 17:04:42,576::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69516::DEBUG::2014-09-25 17:04:42,576::task::1191::Storage.TaskManager.Task::(prepare) Task=`38f23a13-6bcf-473e-9471-d4978ebc775b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69516::DEBUG::2014-09-25 17:04:42,576::task::595::Storage.TaskManager.Task::(_updateState) Task=`38f23a13-6bcf-473e-9471-d4978ebc775b`::moving from state preparing -> state finished >Thread-69516::DEBUG::2014-09-25 17:04:42,577::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69516::DEBUG::2014-09-25 17:04:42,577::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69516::DEBUG::2014-09-25 17:04:42,577::task::993::Storage.TaskManager.Task::(_decref) Task=`38f23a13-6bcf-473e-9471-d4978ebc775b`::ref 0 aborting False >Thread-69516::DEBUG::2014-09-25 17:04:42,577::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69516::DEBUG::2014-09-25 17:04:42,578::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:42,592::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69517::DEBUG::2014-09-25 17:04:42,593::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:04:42,593::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69517::DEBUG::2014-09-25 17:04:42,596::task::595::Storage.TaskManager.Task::(_updateState) Task=`32b078f5-3e6f-4938-a818-6ed8e9b58d1b`::moving from state init -> state preparing >Thread-69517::INFO::2014-09-25 17:04:42,596::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69517::DEBUG::2014-09-25 17:04:42,597::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`377545f0-5900-4c17-a6b6-fdf810080a37`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69517::DEBUG::2014-09-25 17:04:42,597::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69517::DEBUG::2014-09-25 17:04:42,597::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69517::DEBUG::2014-09-25 17:04:42,597::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`377545f0-5900-4c17-a6b6-fdf810080a37`::Granted request >Thread-69517::DEBUG::2014-09-25 17:04:42,598::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`32b078f5-3e6f-4938-a818-6ed8e9b58d1b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69517::DEBUG::2014-09-25 17:04:42,598::task::993::Storage.TaskManager.Task::(_decref) Task=`32b078f5-3e6f-4938-a818-6ed8e9b58d1b`::ref 1 aborting False >Thread-69517::INFO::2014-09-25 17:04:42,603::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69517::DEBUG::2014-09-25 17:04:42,603::task::1191::Storage.TaskManager.Task::(prepare) Task=`32b078f5-3e6f-4938-a818-6ed8e9b58d1b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69517::DEBUG::2014-09-25 17:04:42,603::task::595::Storage.TaskManager.Task::(_updateState) Task=`32b078f5-3e6f-4938-a818-6ed8e9b58d1b`::moving from state preparing -> state finished >Thread-69517::DEBUG::2014-09-25 17:04:42,603::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69517::DEBUG::2014-09-25 17:04:42,604::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69517::DEBUG::2014-09-25 17:04:42,604::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69517::DEBUG::2014-09-25 17:04:42,604::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69517::DEBUG::2014-09-25 17:04:42,605::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69517::DEBUG::2014-09-25 17:04:42,605::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69517::DEBUG::2014-09-25 17:04:42,605::task::993::Storage.TaskManager.Task::(_decref) Task=`32b078f5-3e6f-4938-a818-6ed8e9b58d1b`::ref 0 aborting False >Thread-69517::DEBUG::2014-09-25 17:04:42,606::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69517::DEBUG::2014-09-25 17:04:42,606::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:44,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:44,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199579 s, 51.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:44,479::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:44,480::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:44,480::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:44,480::__init__::232::IOProcess::(_processLogs) (106921) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:44,481::__init__::232::IOProcess::(_processLogs) (106921) Queuing response >Thread-81::DEBUG::2014-09-25 17:04:44,482::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:04:44,491::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00053144 s, 698 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:44,492::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:44,492::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:44,493::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:44,493::__init__::232::IOProcess::(_processLogs) (106922) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:44,493::__init__::232::IOProcess::(_processLogs) (106922) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:44,615::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:44,617::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69518::DEBUG::2014-09-25 17:04:44,619::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:46,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:46,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.023223 s, 44.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:47,634::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:47,636::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69519::DEBUG::2014-09-25 17:04:47,637::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:48,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:48,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154038 s, 66.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:49,212::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:49,212::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:49,212::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:49,212::__init__::232::IOProcess::(_processLogs) (106923) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:49,215::__init__::232::IOProcess::(_processLogs) (106923) Queuing response >Thread-79::DEBUG::2014-09-25 17:04:49,215::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:04:49,233::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00553026 s, 102 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:49,235::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:49,235::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:49,235::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:49,235::__init__::232::IOProcess::(_processLogs) (106924) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:49,240::__init__::232::IOProcess::(_processLogs) (106924) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:49,241::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:49,241::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:49,242::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:49,242::__init__::232::IOProcess::(_processLogs) (106925) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:49,242::__init__::232::IOProcess::(_processLogs) (106925) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:49,243::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:49,243::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:49,244::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:49,244::__init__::232::IOProcess::(_processLogs) (106926) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:49,244::__init__::232::IOProcess::(_processLogs) (106926) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:49,245::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:49,245::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:49,246::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:49,247::__init__::232::IOProcess::(_processLogs) (106927) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:49,247::__init__::232::IOProcess::(_processLogs) (106927) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:50,654::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:50,655::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69520::DEBUG::2014-09-25 17:04:50,657::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:50,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:50,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0208592 s, 49.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:51,598::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:51,598::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:51,599::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:51,599::__init__::232::IOProcess::(_processLogs) (106928) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:51,599::__init__::232::IOProcess::(_processLogs) (106928) Queuing response >Thread-80::DEBUG::2014-09-25 17:04:51,601::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:04:51,610::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000458507 s, 824 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:51,611::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:51,611::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:51,611::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:51,611::__init__::232::IOProcess::(_processLogs) (106929) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:51,612::__init__::232::IOProcess::(_processLogs) (106929) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:52,637::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69521::DEBUG::2014-09-25 17:04:52,638::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:04:52,638::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69521::DEBUG::2014-09-25 17:04:52,640::task::595::Storage.TaskManager.Task::(_updateState) Task=`f4a13734-daf0-47a2-bd8c-8421e26008e1`::moving from state init -> state preparing >Thread-69521::INFO::2014-09-25 17:04:52,641::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69521::INFO::2014-09-25 17:04:52,641::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69521::DEBUG::2014-09-25 17:04:52,641::task::1191::Storage.TaskManager.Task::(prepare) Task=`f4a13734-daf0-47a2-bd8c-8421e26008e1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69521::DEBUG::2014-09-25 17:04:52,642::task::595::Storage.TaskManager.Task::(_updateState) Task=`f4a13734-daf0-47a2-bd8c-8421e26008e1`::moving from state preparing -> state finished >Thread-69521::DEBUG::2014-09-25 17:04:52,642::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69521::DEBUG::2014-09-25 17:04:52,642::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69521::DEBUG::2014-09-25 17:04:52,642::task::993::Storage.TaskManager.Task::(_decref) Task=`f4a13734-daf0-47a2-bd8c-8421e26008e1`::ref 0 aborting False >Thread-69521::DEBUG::2014-09-25 17:04:52,643::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69521::DEBUG::2014-09-25 17:04:52,643::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:52,668::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69522::DEBUG::2014-09-25 17:04:52,669::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:04:52,669::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69522::DEBUG::2014-09-25 17:04:52,671::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a0bfa63-37d0-495b-af9b-93913f0f15ba`::moving from state init -> state preparing >Thread-69522::INFO::2014-09-25 17:04:52,671::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69522::DEBUG::2014-09-25 17:04:52,672::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`dc1041c9-e6ce-4c67-b5d8-dcad03f99145`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69522::DEBUG::2014-09-25 17:04:52,672::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69522::DEBUG::2014-09-25 17:04:52,673::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69522::DEBUG::2014-09-25 17:04:52,673::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`dc1041c9-e6ce-4c67-b5d8-dcad03f99145`::Granted request >Thread-69522::DEBUG::2014-09-25 17:04:52,673::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0a0bfa63-37d0-495b-af9b-93913f0f15ba`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69522::DEBUG::2014-09-25 17:04:52,674::task::993::Storage.TaskManager.Task::(_decref) Task=`0a0bfa63-37d0-495b-af9b-93913f0f15ba`::ref 1 aborting False >Thread-69522::INFO::2014-09-25 17:04:52,678::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69522::DEBUG::2014-09-25 17:04:52,678::task::1191::Storage.TaskManager.Task::(prepare) Task=`0a0bfa63-37d0-495b-af9b-93913f0f15ba`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69522::DEBUG::2014-09-25 17:04:52,678::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a0bfa63-37d0-495b-af9b-93913f0f15ba`::moving from state preparing -> state finished >Thread-69522::DEBUG::2014-09-25 17:04:52,679::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69522::DEBUG::2014-09-25 17:04:52,679::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69522::DEBUG::2014-09-25 17:04:52,679::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69522::DEBUG::2014-09-25 17:04:52,680::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69522::DEBUG::2014-09-25 17:04:52,680::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69522::DEBUG::2014-09-25 17:04:52,680::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69522::DEBUG::2014-09-25 17:04:52,680::task::993::Storage.TaskManager.Task::(_decref) Task=`0a0bfa63-37d0-495b-af9b-93913f0f15ba`::ref 0 aborting False >Thread-69522::DEBUG::2014-09-25 17:04:52,681::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69522::DEBUG::2014-09-25 17:04:52,681::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:52,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:52,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200389 s, 51.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:53,670::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:53,671::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69523::DEBUG::2014-09-25 17:04:53,673::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:04:54,495::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:54,495::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:54,496::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:54,496::__init__::232::IOProcess::(_processLogs) (106930) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:54,520::__init__::232::IOProcess::(_processLogs) (106930) Queuing response >Thread-81::DEBUG::2014-09-25 17:04:54,521::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:04:54,561::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00683714 s, 54.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:54,563::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:54,563::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:54,563::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:54,563::__init__::232::IOProcess::(_processLogs) (106931) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:54,570::__init__::232::IOProcess::(_processLogs) (106931) Queuing response >Dummy-148::DEBUG::2014-09-25 17:04:54,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:54,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136848 s, 74.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:56,686::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:56,688::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69524::DEBUG::2014-09-25 17:04:56,690::task::595::Storage.TaskManager.Task::(_updateState) Task=`bbda5c6e-5e20-48fc-8d2a-38ddc56fa871`::moving from state init -> state preparing >Thread-69524::INFO::2014-09-25 17:04:56,691::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69524::INFO::2014-09-25 17:04:56,691::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00553026', 'lastCheck': '7.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000458507', 'lastCheck': '5.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00683714', 'lastCheck': '2.1', 'valid': True}} >Thread-69524::DEBUG::2014-09-25 17:04:56,692::task::1191::Storage.TaskManager.Task::(prepare) Task=`bbda5c6e-5e20-48fc-8d2a-38ddc56fa871`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00553026', 'lastCheck': '7.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000458507', 'lastCheck': '5.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00683714', 'lastCheck': '2.1', 'valid': True}} >Thread-69524::DEBUG::2014-09-25 17:04:56,692::task::595::Storage.TaskManager.Task::(_updateState) Task=`bbda5c6e-5e20-48fc-8d2a-38ddc56fa871`::moving from state preparing -> state finished >Thread-69524::DEBUG::2014-09-25 17:04:56,692::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69524::DEBUG::2014-09-25 17:04:56,692::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69524::DEBUG::2014-09-25 17:04:56,692::task::993::Storage.TaskManager.Task::(_decref) Task=`bbda5c6e-5e20-48fc-8d2a-38ddc56fa871`::ref 0 aborting False >Thread-69524::DEBUG::2014-09-25 17:04:56,695::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:56,716::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:56,717::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69525::DEBUG::2014-09-25 17:04:56,718::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:04:56,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:56,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0322757 s, 31.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:04:59,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:04:59,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0525195 s, 19.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:59,247::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:59,247::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:59,248::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:59,248::__init__::232::IOProcess::(_processLogs) (106932) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:59,253::__init__::232::IOProcess::(_processLogs) (106932) Queuing response >Thread-79::DEBUG::2014-09-25 17:04:59,254::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:04:59,271::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00086481 s, 650 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:04:59,272::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:59,273::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:59,273::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:59,273::__init__::232::IOProcess::(_processLogs) (106933) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:04:59,276::__init__::232::IOProcess::(_processLogs) (106933) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:59,278::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:59,278::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:59,278::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:59,278::__init__::232::IOProcess::(_processLogs) (106934) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:59,279::__init__::232::IOProcess::(_processLogs) (106934) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:59,279::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:59,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:59,281::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:59,281::__init__::232::IOProcess::(_processLogs) (106935) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:59,281::__init__::232::IOProcess::(_processLogs) (106935) Queuing response >Thread-29::DEBUG::2014-09-25 17:04:59,282::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:04:59,282::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:04:59,283::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:04:59,283::__init__::232::IOProcess::(_processLogs) (106936) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:04:59,284::__init__::232::IOProcess::(_processLogs) (106936) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:04:59,776::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:04:59,778::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69526::DEBUG::2014-09-25 17:04:59,780::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:01,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:01,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0257232 s, 39.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:01,614::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:01,615::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:01,615::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:01,615::__init__::232::IOProcess::(_processLogs) (106937) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:01,616::__init__::232::IOProcess::(_processLogs) (106937) Queuing response >Thread-80::DEBUG::2014-09-25 17:05:01,617::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:05:01,626::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000368452 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:01,627::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:01,628::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:01,628::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:01,628::__init__::232::IOProcess::(_processLogs) (106938) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:01,629::__init__::232::IOProcess::(_processLogs) (106938) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:02,703::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:02,705::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69527::DEBUG::2014-09-25 17:05:02,706::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69527::DEBUG::2014-09-25 17:05:02,708::task::595::Storage.TaskManager.Task::(_updateState) Task=`14a7078b-5158-4ee9-8125-5ad4b8d1bbe2`::moving from state init -> state preparing >Thread-69527::INFO::2014-09-25 17:05:02,708::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69527::INFO::2014-09-25 17:05:02,709::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69527::DEBUG::2014-09-25 17:05:02,709::task::1191::Storage.TaskManager.Task::(prepare) Task=`14a7078b-5158-4ee9-8125-5ad4b8d1bbe2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69527::DEBUG::2014-09-25 17:05:02,709::task::595::Storage.TaskManager.Task::(_updateState) Task=`14a7078b-5158-4ee9-8125-5ad4b8d1bbe2`::moving from state preparing -> state finished >Thread-69527::DEBUG::2014-09-25 17:05:02,710::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69527::DEBUG::2014-09-25 17:05:02,710::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69527::DEBUG::2014-09-25 17:05:02,710::task::993::Storage.TaskManager.Task::(_decref) Task=`14a7078b-5158-4ee9-8125-5ad4b8d1bbe2`::ref 0 aborting False >Thread-69527::DEBUG::2014-09-25 17:05:02,710::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69527::DEBUG::2014-09-25 17:05:02,711::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:02,731::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69528::DEBUG::2014-09-25 17:05:02,732::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:05:02,732::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69528::DEBUG::2014-09-25 17:05:02,734::task::595::Storage.TaskManager.Task::(_updateState) Task=`1aa05ddf-eaf8-42ad-beac-b8dcdec4cc49`::moving from state init -> state preparing >Thread-69528::INFO::2014-09-25 17:05:02,734::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69528::DEBUG::2014-09-25 17:05:02,735::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`036242b8-e9cb-4beb-a76d-866f696f3876`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69528::DEBUG::2014-09-25 17:05:02,735::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69528::DEBUG::2014-09-25 17:05:02,736::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69528::DEBUG::2014-09-25 17:05:02,737::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`036242b8-e9cb-4beb-a76d-866f696f3876`::Granted request >Thread-69528::DEBUG::2014-09-25 17:05:02,737::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1aa05ddf-eaf8-42ad-beac-b8dcdec4cc49`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69528::DEBUG::2014-09-25 17:05:02,738::task::993::Storage.TaskManager.Task::(_decref) Task=`1aa05ddf-eaf8-42ad-beac-b8dcdec4cc49`::ref 1 aborting False >Thread-69528::INFO::2014-09-25 17:05:02,742::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69528::DEBUG::2014-09-25 17:05:02,742::task::1191::Storage.TaskManager.Task::(prepare) Task=`1aa05ddf-eaf8-42ad-beac-b8dcdec4cc49`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69528::DEBUG::2014-09-25 17:05:02,743::task::595::Storage.TaskManager.Task::(_updateState) Task=`1aa05ddf-eaf8-42ad-beac-b8dcdec4cc49`::moving from state preparing -> state finished >Thread-69528::DEBUG::2014-09-25 17:05:02,743::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69528::DEBUG::2014-09-25 17:05:02,743::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69528::DEBUG::2014-09-25 17:05:02,743::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69528::DEBUG::2014-09-25 17:05:02,744::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69528::DEBUG::2014-09-25 17:05:02,744::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69528::DEBUG::2014-09-25 17:05:02,744::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69528::DEBUG::2014-09-25 17:05:02,744::task::993::Storage.TaskManager.Task::(_decref) Task=`1aa05ddf-eaf8-42ad-beac-b8dcdec4cc49`::ref 0 aborting False >Thread-69528::DEBUG::2014-09-25 17:05:02,745::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69528::DEBUG::2014-09-25 17:05:02,745::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:02,799::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:02,800::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69529::DEBUG::2014-09-25 17:05:02,802::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:03,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:03,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184332 s, 55.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:04,571::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:04,572::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:04,572::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:04,572::__init__::232::IOProcess::(_processLogs) (106939) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:04,574::__init__::232::IOProcess::(_processLogs) (106939) Queuing response >Thread-81::DEBUG::2014-09-25 17:05:04,575::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:05:04,599::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00878505 s, 42.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:04,600::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:04,600::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:04,600::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:04,601::__init__::232::IOProcess::(_processLogs) (106940) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:04,607::__init__::232::IOProcess::(_processLogs) (106940) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:05,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:05,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130635 s, 78.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:05,819::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:05,821::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69530::DEBUG::2014-09-25 17:05:05,823::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:07,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:07,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143442 s, 71.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:08,838::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:08,839::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69531::DEBUG::2014-09-25 17:05:08,841::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:05:09,284::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:09,284::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:09,284::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:09,285::__init__::232::IOProcess::(_processLogs) (106941) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:09,302::__init__::232::IOProcess::(_processLogs) (106941) Queuing response >Thread-79::DEBUG::2014-09-25 17:05:09,302::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:05:09,324::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00475537 s, 118 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:09,325::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:09,325::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:09,326::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:09,326::__init__::232::IOProcess::(_processLogs) (106942) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:09,327::__init__::232::IOProcess::(_processLogs) (106942) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:09,329::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:09,329::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:09,330::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:09,330::__init__::232::IOProcess::(_processLogs) (106943) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:09,331::__init__::232::IOProcess::(_processLogs) (106943) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:09,331::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:09,332::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:09,332::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:09,333::__init__::232::IOProcess::(_processLogs) (106944) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:09,333::__init__::232::IOProcess::(_processLogs) (106944) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:09,333::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:09,334::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:09,335::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:09,335::__init__::232::IOProcess::(_processLogs) (106945) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:09,335::__init__::232::IOProcess::(_processLogs) (106945) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:09,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:09,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0217281 s, 47.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:05:11,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:11,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133844 s, 76.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:11,631::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:11,631::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:11,631::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:11,632::__init__::232::IOProcess::(_processLogs) (106946) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:11,632::__init__::232::IOProcess::(_processLogs) (106946) Queuing response >Thread-80::DEBUG::2014-09-25 17:05:11,633::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:05:11,642::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000448334 s, 843 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:11,643::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:11,644::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:11,644::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:11,645::__init__::232::IOProcess::(_processLogs) (106947) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:11,645::__init__::232::IOProcess::(_processLogs) (106947) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:11,854::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:11,856::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69532::DEBUG::2014-09-25 17:05:11,858::task::595::Storage.TaskManager.Task::(_updateState) Task=`ca2bc723-4bcd-4cf7-9d56-0b34df2445ba`::moving from state init -> state preparing >Thread-69532::INFO::2014-09-25 17:05:11,859::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69532::INFO::2014-09-25 17:05:11,859::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00475537', 'lastCheck': '2.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000448334', 'lastCheck': '0.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00878505', 'lastCheck': '7.3', 'valid': True}} >Thread-69532::DEBUG::2014-09-25 17:05:11,860::task::1191::Storage.TaskManager.Task::(prepare) Task=`ca2bc723-4bcd-4cf7-9d56-0b34df2445ba`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00475537', 'lastCheck': '2.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000448334', 'lastCheck': '0.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00878505', 'lastCheck': '7.3', 'valid': True}} >Thread-69532::DEBUG::2014-09-25 17:05:11,860::task::595::Storage.TaskManager.Task::(_updateState) Task=`ca2bc723-4bcd-4cf7-9d56-0b34df2445ba`::moving from state preparing -> state finished >Thread-69532::DEBUG::2014-09-25 17:05:11,860::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69532::DEBUG::2014-09-25 17:05:11,860::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69532::DEBUG::2014-09-25 17:05:11,860::task::993::Storage.TaskManager.Task::(_decref) Task=`ca2bc723-4bcd-4cf7-9d56-0b34df2445ba`::ref 0 aborting False >Thread-69532::DEBUG::2014-09-25 17:05:11,863::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:11,880::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:11,881::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69533::DEBUG::2014-09-25 17:05:11,883::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:12,789::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69534::DEBUG::2014-09-25 17:05:12,790::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:05:12,790::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69534::DEBUG::2014-09-25 17:05:12,793::task::595::Storage.TaskManager.Task::(_updateState) Task=`929466f2-fd6b-42b8-8ef8-b134a59d4b57`::moving from state init -> state preparing >Thread-69534::INFO::2014-09-25 17:05:12,793::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69534::INFO::2014-09-25 17:05:12,793::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69534::DEBUG::2014-09-25 17:05:12,794::task::1191::Storage.TaskManager.Task::(prepare) Task=`929466f2-fd6b-42b8-8ef8-b134a59d4b57`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69534::DEBUG::2014-09-25 17:05:12,794::task::595::Storage.TaskManager.Task::(_updateState) Task=`929466f2-fd6b-42b8-8ef8-b134a59d4b57`::moving from state preparing -> state finished >Thread-69534::DEBUG::2014-09-25 17:05:12,794::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69534::DEBUG::2014-09-25 17:05:12,794::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69534::DEBUG::2014-09-25 17:05:12,795::task::993::Storage.TaskManager.Task::(_decref) Task=`929466f2-fd6b-42b8-8ef8-b134a59d4b57`::ref 0 aborting False >Thread-69534::DEBUG::2014-09-25 17:05:12,795::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69534::DEBUG::2014-09-25 17:05:12,795::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:12,808::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69535::DEBUG::2014-09-25 17:05:12,810::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:05:12,810::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69535::DEBUG::2014-09-25 17:05:12,812::task::595::Storage.TaskManager.Task::(_updateState) Task=`84750c0c-9fc2-4397-b635-979f5fbb527b`::moving from state init -> state preparing >Thread-69535::INFO::2014-09-25 17:05:12,812::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69535::DEBUG::2014-09-25 17:05:12,813::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3c867c06-0d82-47d4-83c3-99e5da4c1d00`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69535::DEBUG::2014-09-25 17:05:12,813::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69535::DEBUG::2014-09-25 17:05:12,813::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69535::DEBUG::2014-09-25 17:05:12,814::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3c867c06-0d82-47d4-83c3-99e5da4c1d00`::Granted request >Thread-69535::DEBUG::2014-09-25 17:05:12,814::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`84750c0c-9fc2-4397-b635-979f5fbb527b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69535::DEBUG::2014-09-25 17:05:12,814::task::993::Storage.TaskManager.Task::(_decref) Task=`84750c0c-9fc2-4397-b635-979f5fbb527b`::ref 1 aborting False >Thread-69535::INFO::2014-09-25 17:05:12,819::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69535::DEBUG::2014-09-25 17:05:12,819::task::1191::Storage.TaskManager.Task::(prepare) Task=`84750c0c-9fc2-4397-b635-979f5fbb527b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69535::DEBUG::2014-09-25 17:05:12,819::task::595::Storage.TaskManager.Task::(_updateState) Task=`84750c0c-9fc2-4397-b635-979f5fbb527b`::moving from state preparing -> state finished >Thread-69535::DEBUG::2014-09-25 17:05:12,820::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69535::DEBUG::2014-09-25 17:05:12,820::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69535::DEBUG::2014-09-25 17:05:12,820::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69535::DEBUG::2014-09-25 17:05:12,820::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69535::DEBUG::2014-09-25 17:05:12,821::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69535::DEBUG::2014-09-25 17:05:12,821::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69535::DEBUG::2014-09-25 17:05:12,821::task::993::Storage.TaskManager.Task::(_decref) Task=`84750c0c-9fc2-4397-b635-979f5fbb527b`::ref 0 aborting False >Thread-69535::DEBUG::2014-09-25 17:05:12,822::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69535::DEBUG::2014-09-25 17:05:12,822::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:13,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:13,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215214 s, 47.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:14,609::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:14,609::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:14,609::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:14,609::__init__::232::IOProcess::(_processLogs) (106948) Got request for method 'statvfs' >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:14,948::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:14,950::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69536::DEBUG::2014-09-25 17:05:14,952::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:05:15,009::__init__::232::IOProcess::(_processLogs) (106948) Queuing response >Thread-81::DEBUG::2014-09-25 17:05:15,010::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:05:15,043::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00824428 s, 45.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:15,044::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:15,045::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:15,045::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:15,045::__init__::232::IOProcess::(_processLogs) (106949) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:15,051::__init__::232::IOProcess::(_processLogs) (106949) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:15,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:15,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131512 s, 77.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:05:17,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:17,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0139304 s, 73.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:17,967::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:17,969::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69537::DEBUG::2014-09-25 17:05:17,971::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:05:19,335::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:19,336::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:19,336::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:19,336::__init__::232::IOProcess::(_processLogs) (106950) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:19,337::__init__::232::IOProcess::(_processLogs) (106950) Queuing response >Thread-79::DEBUG::2014-09-25 17:05:19,338::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:05:19,347::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000436382 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:19,348::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:19,348::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:19,348::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:19,349::__init__::232::IOProcess::(_processLogs) (106951) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:19,349::__init__::232::IOProcess::(_processLogs) (106951) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:19,351::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:19,351::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:19,351::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:19,351::__init__::232::IOProcess::(_processLogs) (106952) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:19,352::__init__::232::IOProcess::(_processLogs) (106952) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:19,352::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:19,353::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:19,353::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:19,353::__init__::232::IOProcess::(_processLogs) (106953) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:19,353::__init__::232::IOProcess::(_processLogs) (106953) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:19,354::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:19,354::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:19,355::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:19,355::__init__::232::IOProcess::(_processLogs) (106954) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:19,355::__init__::232::IOProcess::(_processLogs) (106954) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:19,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:19,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168706 s, 60.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:20,989::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:20,990::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69538::DEBUG::2014-09-25 17:05:20,992::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:05:21,647::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:21,648::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:21,648::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:21,648::__init__::232::IOProcess::(_processLogs) (106955) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:21,649::__init__::232::IOProcess::(_processLogs) (106955) Queuing response >Thread-80::DEBUG::2014-09-25 17:05:21,650::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:05:21,658::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000382938 s, 987 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:21,660::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:21,660::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:21,660::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:21,660::__init__::232::IOProcess::(_processLogs) (106956) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:21,661::__init__::232::IOProcess::(_processLogs) (106956) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:21,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:21,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0241347 s, 42.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:22,859::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69539::DEBUG::2014-09-25 17:05:22,860::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:05:22,860::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69539::DEBUG::2014-09-25 17:05:22,863::task::595::Storage.TaskManager.Task::(_updateState) Task=`6f48fa6f-4e34-4cf2-8843-f01603c4d6e6`::moving from state init -> state preparing >Thread-69539::INFO::2014-09-25 17:05:22,863::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69539::INFO::2014-09-25 17:05:22,864::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69539::DEBUG::2014-09-25 17:05:22,864::task::1191::Storage.TaskManager.Task::(prepare) Task=`6f48fa6f-4e34-4cf2-8843-f01603c4d6e6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69539::DEBUG::2014-09-25 17:05:22,864::task::595::Storage.TaskManager.Task::(_updateState) Task=`6f48fa6f-4e34-4cf2-8843-f01603c4d6e6`::moving from state preparing -> state finished >Thread-69539::DEBUG::2014-09-25 17:05:22,864::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69539::DEBUG::2014-09-25 17:05:22,865::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69539::DEBUG::2014-09-25 17:05:22,865::task::993::Storage.TaskManager.Task::(_decref) Task=`6f48fa6f-4e34-4cf2-8843-f01603c4d6e6`::ref 0 aborting False >Thread-69539::DEBUG::2014-09-25 17:05:22,865::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69539::DEBUG::2014-09-25 17:05:22,866::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:22,879::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:22,881::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69540::DEBUG::2014-09-25 17:05:22,881::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69540::DEBUG::2014-09-25 17:05:22,882::task::595::Storage.TaskManager.Task::(_updateState) Task=`a8f815e3-7b53-429e-ac7f-e99ee085a9a7`::moving from state init -> state preparing >Thread-69540::INFO::2014-09-25 17:05:22,883::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69540::DEBUG::2014-09-25 17:05:22,883::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9d573d10-2de4-4b3d-9f97-39b6e3633e90`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69540::DEBUG::2014-09-25 17:05:22,884::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69540::DEBUG::2014-09-25 17:05:22,884::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69540::DEBUG::2014-09-25 17:05:22,884::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9d573d10-2de4-4b3d-9f97-39b6e3633e90`::Granted request >Thread-69540::DEBUG::2014-09-25 17:05:22,885::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a8f815e3-7b53-429e-ac7f-e99ee085a9a7`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69540::DEBUG::2014-09-25 17:05:22,885::task::993::Storage.TaskManager.Task::(_decref) Task=`a8f815e3-7b53-429e-ac7f-e99ee085a9a7`::ref 1 aborting False >Thread-69540::INFO::2014-09-25 17:05:22,889::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69540::DEBUG::2014-09-25 17:05:22,890::task::1191::Storage.TaskManager.Task::(prepare) Task=`a8f815e3-7b53-429e-ac7f-e99ee085a9a7`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69540::DEBUG::2014-09-25 17:05:22,890::task::595::Storage.TaskManager.Task::(_updateState) Task=`a8f815e3-7b53-429e-ac7f-e99ee085a9a7`::moving from state preparing -> state finished >Thread-69540::DEBUG::2014-09-25 17:05:22,890::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69540::DEBUG::2014-09-25 17:05:22,890::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69540::DEBUG::2014-09-25 17:05:22,891::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69540::DEBUG::2014-09-25 17:05:22,891::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69540::DEBUG::2014-09-25 17:05:22,891::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69540::DEBUG::2014-09-25 17:05:22,891::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69540::DEBUG::2014-09-25 17:05:22,892::task::993::Storage.TaskManager.Task::(_decref) Task=`a8f815e3-7b53-429e-ac7f-e99ee085a9a7`::ref 0 aborting False >Thread-69540::DEBUG::2014-09-25 17:05:22,892::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69540::DEBUG::2014-09-25 17:05:22,893::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:23,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:23,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134352 s, 76.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:24,006::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:24,008::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69541::DEBUG::2014-09-25 17:05:24,010::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:05:25,053::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:25,054::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:25,054::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:25,054::__init__::232::IOProcess::(_processLogs) (106957) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:25,055::__init__::232::IOProcess::(_processLogs) (106957) Queuing response >Thread-81::DEBUG::2014-09-25 17:05:25,056::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:05:25,071::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00511866 s, 72.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:25,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:25,072::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:25,073::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:25,073::__init__::232::IOProcess::(_processLogs) (106958) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:25,079::__init__::232::IOProcess::(_processLogs) (106958) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:25,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:25,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158454 s, 64.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:27,022::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:27,024::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69542::DEBUG::2014-09-25 17:05:27,026::task::595::Storage.TaskManager.Task::(_updateState) Task=`a1f1fbf5-7c84-42a8-a126-4e219de59c50`::moving from state init -> state preparing >Thread-69542::INFO::2014-09-25 17:05:27,027::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69542::INFO::2014-09-25 17:05:27,027::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000436382', 'lastCheck': '7.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000382938', 'lastCheck': '5.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00511866', 'lastCheck': '1.9', 'valid': True}} >Thread-69542::DEBUG::2014-09-25 17:05:27,027::task::1191::Storage.TaskManager.Task::(prepare) Task=`a1f1fbf5-7c84-42a8-a126-4e219de59c50`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000436382', 'lastCheck': '7.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000382938', 'lastCheck': '5.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00511866', 'lastCheck': '1.9', 'valid': True}} >Thread-69542::DEBUG::2014-09-25 17:05:27,028::task::595::Storage.TaskManager.Task::(_updateState) Task=`a1f1fbf5-7c84-42a8-a126-4e219de59c50`::moving from state preparing -> state finished >Thread-69542::DEBUG::2014-09-25 17:05:27,028::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69542::DEBUG::2014-09-25 17:05:27,028::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69542::DEBUG::2014-09-25 17:05:27,028::task::993::Storage.TaskManager.Task::(_decref) Task=`a1f1fbf5-7c84-42a8-a126-4e219de59c50`::ref 0 aborting False >Thread-69542::DEBUG::2014-09-25 17:05:27,031::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:27,048::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:27,049::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69543::DEBUG::2014-09-25 17:05:27,051::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:27,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:27,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193281 s, 53.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:29,357::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:29,357::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:29,357::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:29,358::__init__::232::IOProcess::(_processLogs) (106959) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:29,361::__init__::232::IOProcess::(_processLogs) (106959) Queuing response >Thread-79::DEBUG::2014-09-25 17:05:29,361::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:05:29,380::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00756629 s, 74.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:29,382::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:29,382::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:29,382::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:29,382::__init__::232::IOProcess::(_processLogs) (106960) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:29,385::__init__::232::IOProcess::(_processLogs) (106960) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:29,386::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:29,386::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:29,387::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:29,388::__init__::232::IOProcess::(_processLogs) (106961) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:29,388::__init__::232::IOProcess::(_processLogs) (106961) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:29,388::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:29,389::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:29,389::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:29,389::__init__::232::IOProcess::(_processLogs) (106962) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:29,389::__init__::232::IOProcess::(_processLogs) (106962) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:29,390::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:29,391::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:29,391::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:29,392::__init__::232::IOProcess::(_processLogs) (106963) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:29,392::__init__::232::IOProcess::(_processLogs) (106963) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:29,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:30,112::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:30,114::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69544::DEBUG::2014-09-25 17:05:30,116::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:30,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.22637 s, 4.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:31,664::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:31,664::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:31,664::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:31,665::__init__::232::IOProcess::(_processLogs) (106964) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:31,665::__init__::232::IOProcess::(_processLogs) (106964) Queuing response >Thread-80::DEBUG::2014-09-25 17:05:31,666::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:05:31,675::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000571582 s, 661 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:31,676::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:31,677::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:31,677::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:31,677::__init__::232::IOProcess::(_processLogs) (106965) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:31,678::__init__::232::IOProcess::(_processLogs) (106965) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:32,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:32,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149708 s, 68.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:32,921::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69545::DEBUG::2014-09-25 17:05:32,922::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:05:32,922::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69545::DEBUG::2014-09-25 17:05:32,925::task::595::Storage.TaskManager.Task::(_updateState) Task=`cfaf97be-3086-4f9b-8eba-f98a184012ee`::moving from state init -> state preparing >Thread-69545::INFO::2014-09-25 17:05:32,925::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69545::INFO::2014-09-25 17:05:32,926::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69545::DEBUG::2014-09-25 17:05:32,926::task::1191::Storage.TaskManager.Task::(prepare) Task=`cfaf97be-3086-4f9b-8eba-f98a184012ee`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69545::DEBUG::2014-09-25 17:05:32,926::task::595::Storage.TaskManager.Task::(_updateState) Task=`cfaf97be-3086-4f9b-8eba-f98a184012ee`::moving from state preparing -> state finished >Thread-69545::DEBUG::2014-09-25 17:05:32,926::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69545::DEBUG::2014-09-25 17:05:32,927::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69545::DEBUG::2014-09-25 17:05:32,927::task::993::Storage.TaskManager.Task::(_decref) Task=`cfaf97be-3086-4f9b-8eba-f98a184012ee`::ref 0 aborting False >Thread-69545::DEBUG::2014-09-25 17:05:32,927::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69545::DEBUG::2014-09-25 17:05:32,928::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:32,941::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69546::DEBUG::2014-09-25 17:05:32,942::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:05:32,942::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69546::DEBUG::2014-09-25 17:05:32,944::task::595::Storage.TaskManager.Task::(_updateState) Task=`f085ccb2-054a-4d15-811f-1ebda2a6dbfb`::moving from state init -> state preparing >Thread-69546::INFO::2014-09-25 17:05:32,944::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69546::DEBUG::2014-09-25 17:05:32,945::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8a32ae17-96d3-4d2e-bb21-247ef79eecbc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69546::DEBUG::2014-09-25 17:05:32,945::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69546::DEBUG::2014-09-25 17:05:32,946::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69546::DEBUG::2014-09-25 17:05:32,946::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8a32ae17-96d3-4d2e-bb21-247ef79eecbc`::Granted request >Thread-69546::DEBUG::2014-09-25 17:05:32,947::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f085ccb2-054a-4d15-811f-1ebda2a6dbfb`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69546::DEBUG::2014-09-25 17:05:32,947::task::993::Storage.TaskManager.Task::(_decref) Task=`f085ccb2-054a-4d15-811f-1ebda2a6dbfb`::ref 1 aborting False >Thread-69546::INFO::2014-09-25 17:05:32,951::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69546::DEBUG::2014-09-25 17:05:32,952::task::1191::Storage.TaskManager.Task::(prepare) Task=`f085ccb2-054a-4d15-811f-1ebda2a6dbfb`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69546::DEBUG::2014-09-25 17:05:32,952::task::595::Storage.TaskManager.Task::(_updateState) Task=`f085ccb2-054a-4d15-811f-1ebda2a6dbfb`::moving from state preparing -> state finished >Thread-69546::DEBUG::2014-09-25 17:05:32,952::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69546::DEBUG::2014-09-25 17:05:32,952::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69546::DEBUG::2014-09-25 17:05:32,953::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69546::DEBUG::2014-09-25 17:05:32,953::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69546::DEBUG::2014-09-25 17:05:32,953::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69546::DEBUG::2014-09-25 17:05:32,953::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69546::DEBUG::2014-09-25 17:05:32,954::task::993::Storage.TaskManager.Task::(_decref) Task=`f085ccb2-054a-4d15-811f-1ebda2a6dbfb`::ref 0 aborting False >Thread-69546::DEBUG::2014-09-25 17:05:32,954::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69546::DEBUG::2014-09-25 17:05:32,955::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:33,135::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:33,136::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69547::DEBUG::2014-09-25 17:05:33,138::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:34,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:34,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143877 s, 71.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:35,081::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:35,081::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:35,081::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:35,082::__init__::232::IOProcess::(_processLogs) (106966) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:35,088::__init__::232::IOProcess::(_processLogs) (106966) Queuing response >Thread-81::DEBUG::2014-09-25 17:05:35,089::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:05:35,112::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0071157 s, 52.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:35,113::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:35,113::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:35,113::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:35,114::__init__::232::IOProcess::(_processLogs) (106967) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:35,120::__init__::232::IOProcess::(_processLogs) (106967) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:36,154::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:36,156::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69548::DEBUG::2014-09-25 17:05:36,158::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:36,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:36,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135464 s, 75.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:05:38,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:38,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203877 s, 50.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:39,174::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:39,175::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69549::DEBUG::2014-09-25 17:05:39,177::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:05:39,392::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:39,393::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:39,393::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:39,393::__init__::232::IOProcess::(_processLogs) (106968) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:39,395::__init__::232::IOProcess::(_processLogs) (106968) Queuing response >Thread-79::DEBUG::2014-09-25 17:05:39,396::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:05:39,412::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00315266 s, 178 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:39,414::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:39,414::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:39,414::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:39,414::__init__::232::IOProcess::(_processLogs) (106969) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:39,420::__init__::232::IOProcess::(_processLogs) (106969) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:39,421::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:39,421::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:39,421::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:39,422::__init__::232::IOProcess::(_processLogs) (106970) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:39,426::__init__::232::IOProcess::(_processLogs) (106970) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:39,427::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:39,428::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:39,428::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:39,428::__init__::232::IOProcess::(_processLogs) (106971) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:39,433::__init__::232::IOProcess::(_processLogs) (106971) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:39,434::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:39,435::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:39,435::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:39,435::__init__::232::IOProcess::(_processLogs) (106972) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:39,438::__init__::232::IOProcess::(_processLogs) (106972) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:40,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:40,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.022481 s, 45.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:41,680::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:41,681::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:41,681::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:41,681::__init__::232::IOProcess::(_processLogs) (106973) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:41,682::__init__::232::IOProcess::(_processLogs) (106973) Queuing response >Thread-80::DEBUG::2014-09-25 17:05:41,683::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:05:41,692::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000383717 s, 985 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:41,693::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:41,694::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:41,694::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:41,694::__init__::232::IOProcess::(_processLogs) (106974) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:41,695::__init__::232::IOProcess::(_processLogs) (106974) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:42,194::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:42,196::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69550::DEBUG::2014-09-25 17:05:42,198::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb33c938-1aca-4be9-9a58-a58e1e81f3ac`::moving from state init -> state preparing >Thread-69550::INFO::2014-09-25 17:05:42,199::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69550::INFO::2014-09-25 17:05:42,199::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00315266', 'lastCheck': '2.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000383717', 'lastCheck': '0.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0071157', 'lastCheck': '7.1', 'valid': True}} >Thread-69550::DEBUG::2014-09-25 17:05:42,199::task::1191::Storage.TaskManager.Task::(prepare) Task=`fb33c938-1aca-4be9-9a58-a58e1e81f3ac`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00315266', 'lastCheck': '2.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000383717', 'lastCheck': '0.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0071157', 'lastCheck': '7.1', 'valid': True}} >Thread-69550::DEBUG::2014-09-25 17:05:42,200::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb33c938-1aca-4be9-9a58-a58e1e81f3ac`::moving from state preparing -> state finished >Thread-69550::DEBUG::2014-09-25 17:05:42,200::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69550::DEBUG::2014-09-25 17:05:42,200::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69550::DEBUG::2014-09-25 17:05:42,200::task::993::Storage.TaskManager.Task::(_decref) Task=`fb33c938-1aca-4be9-9a58-a58e1e81f3ac`::ref 0 aborting False >Thread-69550::DEBUG::2014-09-25 17:05:42,203::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:42,224::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:42,225::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69551::DEBUG::2014-09-25 17:05:42,227::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:42,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:42,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160503 s, 63.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:42,987::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69552::DEBUG::2014-09-25 17:05:42,988::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:05:42,988::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69552::DEBUG::2014-09-25 17:05:42,990::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f3ba8ea-22bf-4da1-8756-4fd7de8b8376`::moving from state init -> state preparing >Thread-69552::INFO::2014-09-25 17:05:42,991::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69552::INFO::2014-09-25 17:05:42,991::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69552::DEBUG::2014-09-25 17:05:42,992::task::1191::Storage.TaskManager.Task::(prepare) Task=`0f3ba8ea-22bf-4da1-8756-4fd7de8b8376`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69552::DEBUG::2014-09-25 17:05:42,992::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f3ba8ea-22bf-4da1-8756-4fd7de8b8376`::moving from state preparing -> state finished >Thread-69552::DEBUG::2014-09-25 17:05:42,992::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69552::DEBUG::2014-09-25 17:05:42,992::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69552::DEBUG::2014-09-25 17:05:42,993::task::993::Storage.TaskManager.Task::(_decref) Task=`0f3ba8ea-22bf-4da1-8756-4fd7de8b8376`::ref 0 aborting False >Thread-69552::DEBUG::2014-09-25 17:05:42,993::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69552::DEBUG::2014-09-25 17:05:42,993::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:43,014::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:43,015::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69553::DEBUG::2014-09-25 17:05:43,015::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69553::DEBUG::2014-09-25 17:05:43,017::task::595::Storage.TaskManager.Task::(_updateState) Task=`5f55eac9-30c3-467c-b768-b72a89a0bfb3`::moving from state init -> state preparing >Thread-69553::INFO::2014-09-25 17:05:43,017::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69553::DEBUG::2014-09-25 17:05:43,018::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a284457b-8bd0-464b-ad1b-5c51fd0852b3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69553::DEBUG::2014-09-25 17:05:43,018::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69553::DEBUG::2014-09-25 17:05:43,019::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69553::DEBUG::2014-09-25 17:05:43,019::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a284457b-8bd0-464b-ad1b-5c51fd0852b3`::Granted request >Thread-69553::DEBUG::2014-09-25 17:05:43,019::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5f55eac9-30c3-467c-b768-b72a89a0bfb3`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69553::DEBUG::2014-09-25 17:05:43,020::task::993::Storage.TaskManager.Task::(_decref) Task=`5f55eac9-30c3-467c-b768-b72a89a0bfb3`::ref 1 aborting False >Thread-69553::INFO::2014-09-25 17:05:43,024::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69553::DEBUG::2014-09-25 17:05:43,024::task::1191::Storage.TaskManager.Task::(prepare) Task=`5f55eac9-30c3-467c-b768-b72a89a0bfb3`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69553::DEBUG::2014-09-25 17:05:43,025::task::595::Storage.TaskManager.Task::(_updateState) Task=`5f55eac9-30c3-467c-b768-b72a89a0bfb3`::moving from state preparing -> state finished >Thread-69553::DEBUG::2014-09-25 17:05:43,025::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69553::DEBUG::2014-09-25 17:05:43,025::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69553::DEBUG::2014-09-25 17:05:43,025::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69553::DEBUG::2014-09-25 17:05:43,026::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69553::DEBUG::2014-09-25 17:05:43,026::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69553::DEBUG::2014-09-25 17:05:43,026::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69553::DEBUG::2014-09-25 17:05:43,027::task::993::Storage.TaskManager.Task::(_decref) Task=`5f55eac9-30c3-467c-b768-b72a89a0bfb3`::ref 0 aborting False >Thread-69553::DEBUG::2014-09-25 17:05:43,027::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69553::DEBUG::2014-09-25 17:05:43,027::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:44,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:44,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0242468 s, 42.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:45,123::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:45,123::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:45,124::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:45,124::__init__::232::IOProcess::(_processLogs) (106975) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:45,125::__init__::232::IOProcess::(_processLogs) (106975) Queuing response >Thread-81::DEBUG::2014-09-25 17:05:45,126::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:05:45,142::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00735065 s, 50.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:45,144::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:45,144::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:45,144::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:45,144::__init__::232::IOProcess::(_processLogs) (106976) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:45,146::__init__::232::IOProcess::(_processLogs) (106976) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:45,292::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:45,294::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69554::DEBUG::2014-09-25 17:05:45,296::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:46,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:46,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.024948 s, 41.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:48,314::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:48,315::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69555::DEBUG::2014-09-25 17:05:48,318::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:48,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:48,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016339 s, 62.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:49,440::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:49,440::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:49,440::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:49,441::__init__::232::IOProcess::(_processLogs) (106977) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:49,441::__init__::232::IOProcess::(_processLogs) (106977) Queuing response >Thread-79::DEBUG::2014-09-25 17:05:49,442::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:05:49,451::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000450893 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:49,452::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:49,453::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:49,453::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:49,453::__init__::232::IOProcess::(_processLogs) (106978) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:49,454::__init__::232::IOProcess::(_processLogs) (106978) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:49,455::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:49,456::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:49,456::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:49,457::__init__::232::IOProcess::(_processLogs) (106979) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:49,457::__init__::232::IOProcess::(_processLogs) (106979) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:49,457::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:49,458::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:49,459::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:49,459::__init__::232::IOProcess::(_processLogs) (106980) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:49,459::__init__::232::IOProcess::(_processLogs) (106980) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:49,459::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:49,460::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:49,461::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:49,461::__init__::232::IOProcess::(_processLogs) (106981) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:49,461::__init__::232::IOProcess::(_processLogs) (106981) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:50,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:50,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207219 s, 49.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:51,333::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:51,334::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69556::DEBUG::2014-09-25 17:05:51,336::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:05:51,697::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:51,697::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:51,697::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:51,698::__init__::232::IOProcess::(_processLogs) (106982) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:51,698::__init__::232::IOProcess::(_processLogs) (106982) Queuing response >Thread-80::DEBUG::2014-09-25 17:05:51,699::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:05:51,708::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000395732 s, 955 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:51,709::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:51,710::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:51,710::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:51,710::__init__::232::IOProcess::(_processLogs) (106983) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:51,711::__init__::232::IOProcess::(_processLogs) (106983) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:52,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:52,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135108 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:53,064::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69557::DEBUG::2014-09-25 17:05:53,065::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:05:53,065::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69557::DEBUG::2014-09-25 17:05:53,068::task::595::Storage.TaskManager.Task::(_updateState) Task=`fa3be6a9-7c21-49b9-b252-972fe8eb116d`::moving from state init -> state preparing >Thread-69557::INFO::2014-09-25 17:05:53,068::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69557::INFO::2014-09-25 17:05:53,068::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69557::DEBUG::2014-09-25 17:05:53,069::task::1191::Storage.TaskManager.Task::(prepare) Task=`fa3be6a9-7c21-49b9-b252-972fe8eb116d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69557::DEBUG::2014-09-25 17:05:53,069::task::595::Storage.TaskManager.Task::(_updateState) Task=`fa3be6a9-7c21-49b9-b252-972fe8eb116d`::moving from state preparing -> state finished >Thread-69557::DEBUG::2014-09-25 17:05:53,069::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69557::DEBUG::2014-09-25 17:05:53,069::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69557::DEBUG::2014-09-25 17:05:53,070::task::993::Storage.TaskManager.Task::(_decref) Task=`fa3be6a9-7c21-49b9-b252-972fe8eb116d`::ref 0 aborting False >Thread-69557::DEBUG::2014-09-25 17:05:53,070::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69557::DEBUG::2014-09-25 17:05:53,070::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:53,084::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:53,085::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69558::DEBUG::2014-09-25 17:05:53,085::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69558::DEBUG::2014-09-25 17:05:53,087::task::595::Storage.TaskManager.Task::(_updateState) Task=`ffaa6fa4-b0c6-443d-998b-06338f820e9b`::moving from state init -> state preparing >Thread-69558::INFO::2014-09-25 17:05:53,087::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69558::DEBUG::2014-09-25 17:05:53,088::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ab3e9913-0d67-4136-b684-1ff17c4955cc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69558::DEBUG::2014-09-25 17:05:53,088::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69558::DEBUG::2014-09-25 17:05:53,088::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69558::DEBUG::2014-09-25 17:05:53,089::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ab3e9913-0d67-4136-b684-1ff17c4955cc`::Granted request >Thread-69558::DEBUG::2014-09-25 17:05:53,089::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ffaa6fa4-b0c6-443d-998b-06338f820e9b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69558::DEBUG::2014-09-25 17:05:53,090::task::993::Storage.TaskManager.Task::(_decref) Task=`ffaa6fa4-b0c6-443d-998b-06338f820e9b`::ref 1 aborting False >Thread-69558::INFO::2014-09-25 17:05:53,094::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69558::DEBUG::2014-09-25 17:05:53,094::task::1191::Storage.TaskManager.Task::(prepare) Task=`ffaa6fa4-b0c6-443d-998b-06338f820e9b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69558::DEBUG::2014-09-25 17:05:53,094::task::595::Storage.TaskManager.Task::(_updateState) Task=`ffaa6fa4-b0c6-443d-998b-06338f820e9b`::moving from state preparing -> state finished >Thread-69558::DEBUG::2014-09-25 17:05:53,095::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69558::DEBUG::2014-09-25 17:05:53,095::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69558::DEBUG::2014-09-25 17:05:53,095::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69558::DEBUG::2014-09-25 17:05:53,096::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69558::DEBUG::2014-09-25 17:05:53,096::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69558::DEBUG::2014-09-25 17:05:53,096::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69558::DEBUG::2014-09-25 17:05:53,096::task::993::Storage.TaskManager.Task::(_decref) Task=`ffaa6fa4-b0c6-443d-998b-06338f820e9b`::ref 0 aborting False >Thread-69558::DEBUG::2014-09-25 17:05:53,097::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69558::DEBUG::2014-09-25 17:05:53,097::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:54,351::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:54,352::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69559::DEBUG::2014-09-25 17:05:54,354::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:54,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:54,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164448 s, 62.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:55,148::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:55,149::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:55,149::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:55,149::__init__::232::IOProcess::(_processLogs) (106984) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:55,176::__init__::232::IOProcess::(_processLogs) (106984) Queuing response >Thread-81::DEBUG::2014-09-25 17:05:55,177::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:05:55,227::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00491111 s, 75.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:55,228::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:55,229::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:55,229::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:55,229::__init__::232::IOProcess::(_processLogs) (106985) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:55,235::__init__::232::IOProcess::(_processLogs) (106985) Queuing response >Dummy-148::DEBUG::2014-09-25 17:05:56,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:56,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142214 s, 72.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:57,369::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:57,370::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69560::DEBUG::2014-09-25 17:05:57,373::task::595::Storage.TaskManager.Task::(_updateState) Task=`8fc5f2e5-0a96-403a-a83d-5cb0c1df6107`::moving from state init -> state preparing >Thread-69560::INFO::2014-09-25 17:05:57,373::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69560::INFO::2014-09-25 17:05:57,374::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000450893', 'lastCheck': '7.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000395732', 'lastCheck': '5.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00491111', 'lastCheck': '2.1', 'valid': True}} >Thread-69560::DEBUG::2014-09-25 17:05:57,374::task::1191::Storage.TaskManager.Task::(prepare) Task=`8fc5f2e5-0a96-403a-a83d-5cb0c1df6107`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000450893', 'lastCheck': '7.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000395732', 'lastCheck': '5.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00491111', 'lastCheck': '2.1', 'valid': True}} >Thread-69560::DEBUG::2014-09-25 17:05:57,374::task::595::Storage.TaskManager.Task::(_updateState) Task=`8fc5f2e5-0a96-403a-a83d-5cb0c1df6107`::moving from state preparing -> state finished >Thread-69560::DEBUG::2014-09-25 17:05:57,375::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69560::DEBUG::2014-09-25 17:05:57,375::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69560::DEBUG::2014-09-25 17:05:57,375::task::993::Storage.TaskManager.Task::(_decref) Task=`8fc5f2e5-0a96-403a-a83d-5cb0c1df6107`::ref 0 aborting False >Thread-69560::DEBUG::2014-09-25 17:05:57,377::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:05:57,399::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:05:57,400::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69561::DEBUG::2014-09-25 17:05:57,402::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:05:58,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:05:59,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0242799 s, 42.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:59,462::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:59,462::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:59,462::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:59,462::__init__::232::IOProcess::(_processLogs) (106986) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:59,468::__init__::232::IOProcess::(_processLogs) (106986) Queuing response >Thread-79::DEBUG::2014-09-25 17:05:59,469::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:05:59,500::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00782967 s, 71.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:05:59,501::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:59,502::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:59,502::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:59,502::__init__::232::IOProcess::(_processLogs) (106987) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:05:59,503::__init__::232::IOProcess::(_processLogs) (106987) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:59,505::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:59,505::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:59,505::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:59,506::__init__::232::IOProcess::(_processLogs) (106988) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:59,506::__init__::232::IOProcess::(_processLogs) (106988) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:59,507::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:59,507::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:59,507::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:59,508::__init__::232::IOProcess::(_processLogs) (106989) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:59,508::__init__::232::IOProcess::(_processLogs) (106989) Queuing response >Thread-29::DEBUG::2014-09-25 17:05:59,509::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:05:59,509::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:05:59,509::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:05:59,511::__init__::232::IOProcess::(_processLogs) (106990) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:05:59,511::__init__::232::IOProcess::(_processLogs) (106990) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:00,470::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:00,472::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69562::DEBUG::2014-09-25 17:06:00,473::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:01,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:01,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148008 s, 69.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:01,713::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:01,713::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:01,714::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:06:01,715::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:06:01,715::__init__::232::IOProcess::(_processLogs) (106991) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:01,724::__init__::232::IOProcess::(_processLogs) (106991) Queuing response >Thread-80::DEBUG::2014-09-25 17:06:01,725::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00045057 s, 839 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:01,726::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:01,727::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:01,727::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:01,728::__init__::232::IOProcess::(_processLogs) (106992) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:01,729::__init__::232::IOProcess::(_processLogs) (106992) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:03,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:03,130::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:03,134::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69563::DEBUG::2014-09-25 17:06:03,134::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Dummy-148::DEBUG::2014-09-25 17:06:03,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0266213 s, 38.5 MB/s\n'; <rc> = 0 >Thread-69563::DEBUG::2014-09-25 17:06:03,139::task::595::Storage.TaskManager.Task::(_updateState) Task=`4448a125-dbf2-4973-bb09-34058ad18c95`::moving from state init -> state preparing >Thread-69563::INFO::2014-09-25 17:06:03,140::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69563::INFO::2014-09-25 17:06:03,142::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69563::DEBUG::2014-09-25 17:06:03,143::task::1191::Storage.TaskManager.Task::(prepare) Task=`4448a125-dbf2-4973-bb09-34058ad18c95`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69563::DEBUG::2014-09-25 17:06:03,144::task::595::Storage.TaskManager.Task::(_updateState) Task=`4448a125-dbf2-4973-bb09-34058ad18c95`::moving from state preparing -> state finished >Thread-69563::DEBUG::2014-09-25 17:06:03,145::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69563::DEBUG::2014-09-25 17:06:03,145::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69563::DEBUG::2014-09-25 17:06:03,146::task::993::Storage.TaskManager.Task::(_decref) Task=`4448a125-dbf2-4973-bb09-34058ad18c95`::ref 0 aborting False >Thread-69563::DEBUG::2014-09-25 17:06:03,147::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69563::DEBUG::2014-09-25 17:06:03,149::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:03,169::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:03,170::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69564::DEBUG::2014-09-25 17:06:03,170::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69564::DEBUG::2014-09-25 17:06:03,172::task::595::Storage.TaskManager.Task::(_updateState) Task=`bad928fa-4067-47e2-97b7-2b2df6ac3ad8`::moving from state init -> state preparing >Thread-69564::INFO::2014-09-25 17:06:03,172::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69564::DEBUG::2014-09-25 17:06:03,173::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7ac1f5ed-54f7-45b4-a26d-c998f6546af5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69564::DEBUG::2014-09-25 17:06:03,173::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69564::DEBUG::2014-09-25 17:06:03,173::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69564::DEBUG::2014-09-25 17:06:03,174::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7ac1f5ed-54f7-45b4-a26d-c998f6546af5`::Granted request >Thread-69564::DEBUG::2014-09-25 17:06:03,174::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`bad928fa-4067-47e2-97b7-2b2df6ac3ad8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69564::DEBUG::2014-09-25 17:06:03,174::task::993::Storage.TaskManager.Task::(_decref) Task=`bad928fa-4067-47e2-97b7-2b2df6ac3ad8`::ref 1 aborting False >Thread-69564::INFO::2014-09-25 17:06:03,179::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69564::DEBUG::2014-09-25 17:06:03,179::task::1191::Storage.TaskManager.Task::(prepare) Task=`bad928fa-4067-47e2-97b7-2b2df6ac3ad8`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69564::DEBUG::2014-09-25 17:06:03,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`bad928fa-4067-47e2-97b7-2b2df6ac3ad8`::moving from state preparing -> state finished >Thread-69564::DEBUG::2014-09-25 17:06:03,180::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69564::DEBUG::2014-09-25 17:06:03,180::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69564::DEBUG::2014-09-25 17:06:03,180::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69564::DEBUG::2014-09-25 17:06:03,181::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69564::DEBUG::2014-09-25 17:06:03,181::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69564::DEBUG::2014-09-25 17:06:03,181::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69564::DEBUG::2014-09-25 17:06:03,181::task::993::Storage.TaskManager.Task::(_decref) Task=`bad928fa-4067-47e2-97b7-2b2df6ac3ad8`::ref 0 aborting False >Thread-69564::DEBUG::2014-09-25 17:06:03,182::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69564::DEBUG::2014-09-25 17:06:03,182::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:03,489::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:03,491::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69565::DEBUG::2014-09-25 17:06:03,492::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:05,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:05,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132854 s, 77.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:05,237::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:05,238::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:05,238::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:05,238::__init__::232::IOProcess::(_processLogs) (106993) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:05,238::__init__::232::IOProcess::(_processLogs) (106993) Queuing response >Thread-81::DEBUG::2014-09-25 17:06:05,239::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:06:05,248::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000418895 s, 886 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:05,249::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:05,249::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:05,250::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:05,250::__init__::232::IOProcess::(_processLogs) (106994) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:05,250::__init__::232::IOProcess::(_processLogs) (106994) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:06,507::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:06,508::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69566::DEBUG::2014-09-25 17:06:06,510::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:07,211::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:07,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134026 s, 76.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:06:09,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:09,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0221722 s, 46.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:09,511::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:09,512::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:09,512::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:09,512::__init__::232::IOProcess::(_processLogs) (106995) Got request for method 'statvfs' >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:09,526::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:09,527::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69567::DEBUG::2014-09-25 17:06:09,530::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:06:09,534::__init__::232::IOProcess::(_processLogs) (106995) Queuing response >Thread-79::DEBUG::2014-09-25 17:06:09,535::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:06:09,567::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00884768 s, 63.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:09,568::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:09,569::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:09,569::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:09,569::__init__::232::IOProcess::(_processLogs) (106996) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:09,573::__init__::232::IOProcess::(_processLogs) (106996) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:09,574::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:09,574::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:09,575::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:09,575::__init__::232::IOProcess::(_processLogs) (106997) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:09,576::__init__::232::IOProcess::(_processLogs) (106997) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:09,576::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:09,576::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:09,577::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:09,577::__init__::232::IOProcess::(_processLogs) (106998) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:09,578::__init__::232::IOProcess::(_processLogs) (106998) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:09,578::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:09,579::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:09,579::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:09,579::__init__::232::IOProcess::(_processLogs) (106999) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:09,579::__init__::232::IOProcess::(_processLogs) (106999) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:11,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:11,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0198422 s, 51.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:11,729::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:11,730::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:11,730::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:11,730::__init__::232::IOProcess::(_processLogs) (107000) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:11,730::__init__::232::IOProcess::(_processLogs) (107000) Queuing response >Thread-80::DEBUG::2014-09-25 17:06:11,732::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:06:11,740::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000519743 s, 727 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:11,741::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:11,742::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:11,743::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:11,744::__init__::232::IOProcess::(_processLogs) (107001) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:11,744::__init__::232::IOProcess::(_processLogs) (107001) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:12,544::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:12,545::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69568::DEBUG::2014-09-25 17:06:12,548::task::595::Storage.TaskManager.Task::(_updateState) Task=`c15096de-1c0b-44f7-b8b9-7b1265b20155`::moving from state init -> state preparing >Thread-69568::INFO::2014-09-25 17:06:12,548::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69568::INFO::2014-09-25 17:06:12,549::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00884768', 'lastCheck': '3.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000519743', 'lastCheck': '0.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000418895', 'lastCheck': '7.3', 'valid': True}} >Thread-69568::DEBUG::2014-09-25 17:06:12,549::task::1191::Storage.TaskManager.Task::(prepare) Task=`c15096de-1c0b-44f7-b8b9-7b1265b20155`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00884768', 'lastCheck': '3.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000519743', 'lastCheck': '0.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000418895', 'lastCheck': '7.3', 'valid': True}} >Thread-69568::DEBUG::2014-09-25 17:06:12,549::task::595::Storage.TaskManager.Task::(_updateState) Task=`c15096de-1c0b-44f7-b8b9-7b1265b20155`::moving from state preparing -> state finished >Thread-69568::DEBUG::2014-09-25 17:06:12,549::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69568::DEBUG::2014-09-25 17:06:12,549::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69568::DEBUG::2014-09-25 17:06:12,550::task::993::Storage.TaskManager.Task::(_decref) Task=`c15096de-1c0b-44f7-b8b9-7b1265b20155`::ref 0 aborting False >Thread-69568::DEBUG::2014-09-25 17:06:12,552::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:12,573::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:12,574::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69569::DEBUG::2014-09-25 17:06:12,576::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:13,211::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69570::DEBUG::2014-09-25 17:06:13,212::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:06:13,212::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69570::DEBUG::2014-09-25 17:06:13,214::task::595::Storage.TaskManager.Task::(_updateState) Task=`972ca02f-f4f2-4fd4-b129-ff1b7a722511`::moving from state init -> state preparing >Thread-69570::INFO::2014-09-25 17:06:13,215::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69570::INFO::2014-09-25 17:06:13,215::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69570::DEBUG::2014-09-25 17:06:13,216::task::1191::Storage.TaskManager.Task::(prepare) Task=`972ca02f-f4f2-4fd4-b129-ff1b7a722511`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69570::DEBUG::2014-09-25 17:06:13,216::task::595::Storage.TaskManager.Task::(_updateState) Task=`972ca02f-f4f2-4fd4-b129-ff1b7a722511`::moving from state preparing -> state finished >Thread-69570::DEBUG::2014-09-25 17:06:13,216::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69570::DEBUG::2014-09-25 17:06:13,216::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69570::DEBUG::2014-09-25 17:06:13,216::task::993::Storage.TaskManager.Task::(_decref) Task=`972ca02f-f4f2-4fd4-b129-ff1b7a722511`::ref 0 aborting False >Thread-69570::DEBUG::2014-09-25 17:06:13,217::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69570::DEBUG::2014-09-25 17:06:13,217::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:13,231::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69571::DEBUG::2014-09-25 17:06:13,232::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:06:13,232::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69571::DEBUG::2014-09-25 17:06:13,234::task::595::Storage.TaskManager.Task::(_updateState) Task=`f050ca8c-e9b1-445d-a95c-e5f9bd09b61f`::moving from state init -> state preparing >Thread-69571::INFO::2014-09-25 17:06:13,235::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69571::DEBUG::2014-09-25 17:06:13,235::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5422ae84-3825-415b-ae62-8a1f059bd829`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69571::DEBUG::2014-09-25 17:06:13,235::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69571::DEBUG::2014-09-25 17:06:13,236::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69571::DEBUG::2014-09-25 17:06:13,236::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5422ae84-3825-415b-ae62-8a1f059bd829`::Granted request >Thread-69571::DEBUG::2014-09-25 17:06:13,237::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f050ca8c-e9b1-445d-a95c-e5f9bd09b61f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69571::DEBUG::2014-09-25 17:06:13,237::task::993::Storage.TaskManager.Task::(_decref) Task=`f050ca8c-e9b1-445d-a95c-e5f9bd09b61f`::ref 1 aborting False >Thread-69571::INFO::2014-09-25 17:06:13,241::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69571::DEBUG::2014-09-25 17:06:13,242::task::1191::Storage.TaskManager.Task::(prepare) Task=`f050ca8c-e9b1-445d-a95c-e5f9bd09b61f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69571::DEBUG::2014-09-25 17:06:13,242::task::595::Storage.TaskManager.Task::(_updateState) Task=`f050ca8c-e9b1-445d-a95c-e5f9bd09b61f`::moving from state preparing -> state finished >Thread-69571::DEBUG::2014-09-25 17:06:13,242::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69571::DEBUG::2014-09-25 17:06:13,243::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69571::DEBUG::2014-09-25 17:06:13,243::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69571::DEBUG::2014-09-25 17:06:13,243::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69571::DEBUG::2014-09-25 17:06:13,243::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69571::DEBUG::2014-09-25 17:06:13,244::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69571::DEBUG::2014-09-25 17:06:13,244::task::993::Storage.TaskManager.Task::(_decref) Task=`f050ca8c-e9b1-445d-a95c-e5f9bd09b61f`::ref 0 aborting False >Thread-69571::DEBUG::2014-09-25 17:06:13,244::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69571::DEBUG::2014-09-25 17:06:13,245::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:13,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:13,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168977 s, 60.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:15,252::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:15,252::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:15,253::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:15,253::__init__::232::IOProcess::(_processLogs) (107002) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:15,260::__init__::232::IOProcess::(_processLogs) (107002) Queuing response >Thread-81::DEBUG::2014-09-25 17:06:15,261::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:06:15,294::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00896225 s, 41.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:15,295::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:15,295::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:15,296::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:15,296::__init__::232::IOProcess::(_processLogs) (107003) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:15,302::__init__::232::IOProcess::(_processLogs) (107003) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:15,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:15,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133117 s, 76.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:15,640::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:15,642::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69572::DEBUG::2014-09-25 17:06:15,644::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:17,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:17,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0224089 s, 45.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:18,660::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:18,662::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69573::DEBUG::2014-09-25 17:06:18,664::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:19,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:19,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143823 s, 71.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:19,581::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:19,581::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:19,581::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:19,582::__init__::232::IOProcess::(_processLogs) (107004) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:19,585::__init__::232::IOProcess::(_processLogs) (107004) Queuing response >Thread-79::DEBUG::2014-09-25 17:06:19,586::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:06:19,612::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0101073 s, 55.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:19,613::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:19,613::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:19,614::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:19,614::__init__::232::IOProcess::(_processLogs) (107005) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:19,618::__init__::232::IOProcess::(_processLogs) (107005) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:19,619::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:19,619::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:19,620::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:19,621::__init__::232::IOProcess::(_processLogs) (107006) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:19,621::__init__::232::IOProcess::(_processLogs) (107006) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:19,621::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:19,622::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:19,622::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:19,622::__init__::232::IOProcess::(_processLogs) (107007) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:19,622::__init__::232::IOProcess::(_processLogs) (107007) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:19,624::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:19,624::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:19,624::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:19,625::__init__::232::IOProcess::(_processLogs) (107008) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:19,626::__init__::232::IOProcess::(_processLogs) (107008) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:21,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:21,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137042 s, 74.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:21,677::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:21,679::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69574::DEBUG::2014-09-25 17:06:21,681::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:06:21,744::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:21,744::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:21,745::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:21,745::__init__::232::IOProcess::(_processLogs) (107009) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:21,745::__init__::232::IOProcess::(_processLogs) (107009) Queuing response >Thread-80::DEBUG::2014-09-25 17:06:21,747::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:06:21,755::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000506936 s, 746 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:21,756::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:21,757::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:21,757::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:21,757::__init__::232::IOProcess::(_processLogs) (107010) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:21,758::__init__::232::IOProcess::(_processLogs) (107010) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:23,275::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69575::DEBUG::2014-09-25 17:06:23,276::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:06:23,276::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69575::DEBUG::2014-09-25 17:06:23,278::task::595::Storage.TaskManager.Task::(_updateState) Task=`1df12b10-949e-4334-baf5-ee7e8baafdfd`::moving from state init -> state preparing >Thread-69575::INFO::2014-09-25 17:06:23,279::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69575::INFO::2014-09-25 17:06:23,279::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69575::DEBUG::2014-09-25 17:06:23,279::task::1191::Storage.TaskManager.Task::(prepare) Task=`1df12b10-949e-4334-baf5-ee7e8baafdfd`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69575::DEBUG::2014-09-25 17:06:23,280::task::595::Storage.TaskManager.Task::(_updateState) Task=`1df12b10-949e-4334-baf5-ee7e8baafdfd`::moving from state preparing -> state finished >Thread-69575::DEBUG::2014-09-25 17:06:23,280::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69575::DEBUG::2014-09-25 17:06:23,280::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69575::DEBUG::2014-09-25 17:06:23,280::task::993::Storage.TaskManager.Task::(_decref) Task=`1df12b10-949e-4334-baf5-ee7e8baafdfd`::ref 0 aborting False >Thread-69575::DEBUG::2014-09-25 17:06:23,281::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69575::DEBUG::2014-09-25 17:06:23,281::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:23,294::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69576::DEBUG::2014-09-25 17:06:23,295::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:06:23,295::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69576::DEBUG::2014-09-25 17:06:23,297::task::595::Storage.TaskManager.Task::(_updateState) Task=`314448f0-9586-4553-8531-185eed6daef9`::moving from state init -> state preparing >Thread-69576::INFO::2014-09-25 17:06:23,298::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69576::DEBUG::2014-09-25 17:06:23,298::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3257237a-3616-488b-8303-7950f332c251`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69576::DEBUG::2014-09-25 17:06:23,299::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69576::DEBUG::2014-09-25 17:06:23,299::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69576::DEBUG::2014-09-25 17:06:23,299::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3257237a-3616-488b-8303-7950f332c251`::Granted request >Thread-69576::DEBUG::2014-09-25 17:06:23,300::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`314448f0-9586-4553-8531-185eed6daef9`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69576::DEBUG::2014-09-25 17:06:23,300::task::993::Storage.TaskManager.Task::(_decref) Task=`314448f0-9586-4553-8531-185eed6daef9`::ref 1 aborting False >Thread-69576::INFO::2014-09-25 17:06:23,304::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69576::DEBUG::2014-09-25 17:06:23,305::task::1191::Storage.TaskManager.Task::(prepare) Task=`314448f0-9586-4553-8531-185eed6daef9`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69576::DEBUG::2014-09-25 17:06:23,305::task::595::Storage.TaskManager.Task::(_updateState) Task=`314448f0-9586-4553-8531-185eed6daef9`::moving from state preparing -> state finished >Thread-69576::DEBUG::2014-09-25 17:06:23,305::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69576::DEBUG::2014-09-25 17:06:23,305::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69576::DEBUG::2014-09-25 17:06:23,306::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69576::DEBUG::2014-09-25 17:06:23,306::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69576::DEBUG::2014-09-25 17:06:23,306::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69576::DEBUG::2014-09-25 17:06:23,307::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69576::DEBUG::2014-09-25 17:06:23,307::task::993::Storage.TaskManager.Task::(_decref) Task=`314448f0-9586-4553-8531-185eed6daef9`::ref 0 aborting False >Thread-69576::DEBUG::2014-09-25 17:06:23,307::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69576::DEBUG::2014-09-25 17:06:23,308::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:23,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:23,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180985 s, 56.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:24,695::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:24,696::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69577::DEBUG::2014-09-25 17:06:24,698::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:06:25,304::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:25,304::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:25,304::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:25,305::__init__::232::IOProcess::(_processLogs) (107011) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:25,306::__init__::232::IOProcess::(_processLogs) (107011) Queuing response >Thread-81::DEBUG::2014-09-25 17:06:25,307::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:06:25,315::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000412005 s, 900 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:25,316::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:25,317::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:25,317::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:25,317::__init__::232::IOProcess::(_processLogs) (107012) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:25,318::__init__::232::IOProcess::(_processLogs) (107012) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:25,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:25,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191536 s, 53.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:06:27,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:27,710::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:27,712::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69578::DEBUG::2014-09-25 17:06:27,717::task::595::Storage.TaskManager.Task::(_updateState) Task=`8bc4707a-f9fb-4657-a239-a713316413f0`::moving from state init -> state preparing >Thread-69578::INFO::2014-09-25 17:06:27,718::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69578::INFO::2014-09-25 17:06:27,720::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0101073', 'lastCheck': '8.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000506936', 'lastCheck': '6.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000412005', 'lastCheck': '2.4', 'valid': True}} >Thread-69578::DEBUG::2014-09-25 17:06:27,721::task::1191::Storage.TaskManager.Task::(prepare) Task=`8bc4707a-f9fb-4657-a239-a713316413f0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0101073', 'lastCheck': '8.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000506936', 'lastCheck': '6.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000412005', 'lastCheck': '2.4', 'valid': True}} >Thread-69578::DEBUG::2014-09-25 17:06:27,722::task::595::Storage.TaskManager.Task::(_updateState) Task=`8bc4707a-f9fb-4657-a239-a713316413f0`::moving from state preparing -> state finished >Thread-69578::DEBUG::2014-09-25 17:06:27,723::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69578::DEBUG::2014-09-25 17:06:27,723::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69578::DEBUG::2014-09-25 17:06:27,724::task::993::Storage.TaskManager.Task::(_decref) Task=`8bc4707a-f9fb-4657-a239-a713316413f0`::ref 0 aborting False >Thread-69578::DEBUG::2014-09-25 17:06:27,732::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:27,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0326194 s, 31.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:27,758::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:27,759::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69579::DEBUG::2014-09-25 17:06:27,761::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:06:29,626::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:29,627::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:29,627::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:29,627::__init__::232::IOProcess::(_processLogs) (107013) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:29,631::__init__::232::IOProcess::(_processLogs) (107013) Queuing response >Thread-79::DEBUG::2014-09-25 17:06:29,633::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:06:29,650::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00521283 s, 108 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:29,651::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:29,651::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:29,651::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:29,652::__init__::232::IOProcess::(_processLogs) (107014) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:29,656::__init__::232::IOProcess::(_processLogs) (107014) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:29,658::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:29,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:29,659::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:29,659::__init__::232::IOProcess::(_processLogs) (107015) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:29,659::__init__::232::IOProcess::(_processLogs) (107015) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:29,660::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:29,660::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:29,661::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:29,661::__init__::232::IOProcess::(_processLogs) (107016) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:29,661::__init__::232::IOProcess::(_processLogs) (107016) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:29,662::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:29,662::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:29,663::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:29,664::__init__::232::IOProcess::(_processLogs) (107017) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:29,664::__init__::232::IOProcess::(_processLogs) (107017) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:29,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:29,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016877 s, 60.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:30,811::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:30,814::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69580::DEBUG::2014-09-25 17:06:30,816::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:06:31,760::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:31,760::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:31,760::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:31,761::__init__::232::IOProcess::(_processLogs) (107018) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:31,761::__init__::232::IOProcess::(_processLogs) (107018) Queuing response >Thread-80::DEBUG::2014-09-25 17:06:31,762::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:06:31,772::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000481018 s, 786 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:31,773::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:31,773::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:31,774::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:31,774::__init__::232::IOProcess::(_processLogs) (107019) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:31,774::__init__::232::IOProcess::(_processLogs) (107019) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:31,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:31,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183254 s, 55.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:33,330::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69581::DEBUG::2014-09-25 17:06:33,331::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:06:33,331::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69581::DEBUG::2014-09-25 17:06:33,334::task::595::Storage.TaskManager.Task::(_updateState) Task=`badb44b8-359a-4d87-8a11-59ced48c50a2`::moving from state init -> state preparing >Thread-69581::INFO::2014-09-25 17:06:33,334::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69581::INFO::2014-09-25 17:06:33,335::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69581::DEBUG::2014-09-25 17:06:33,335::task::1191::Storage.TaskManager.Task::(prepare) Task=`badb44b8-359a-4d87-8a11-59ced48c50a2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69581::DEBUG::2014-09-25 17:06:33,335::task::595::Storage.TaskManager.Task::(_updateState) Task=`badb44b8-359a-4d87-8a11-59ced48c50a2`::moving from state preparing -> state finished >Thread-69581::DEBUG::2014-09-25 17:06:33,335::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69581::DEBUG::2014-09-25 17:06:33,336::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69581::DEBUG::2014-09-25 17:06:33,336::task::993::Storage.TaskManager.Task::(_decref) Task=`badb44b8-359a-4d87-8a11-59ced48c50a2`::ref 0 aborting False >Thread-69581::DEBUG::2014-09-25 17:06:33,336::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69581::DEBUG::2014-09-25 17:06:33,337::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:33,346::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69582::DEBUG::2014-09-25 17:06:33,348::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:06:33,348::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69582::DEBUG::2014-09-25 17:06:33,350::task::595::Storage.TaskManager.Task::(_updateState) Task=`07cb7b8c-4e8a-4c8b-b07b-3e304e560de3`::moving from state init -> state preparing >Thread-69582::INFO::2014-09-25 17:06:33,350::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69582::DEBUG::2014-09-25 17:06:33,351::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`eb5d29f7-a8a1-480c-b0d7-c7dcd55dcd2a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69582::DEBUG::2014-09-25 17:06:33,351::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69582::DEBUG::2014-09-25 17:06:33,351::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69582::DEBUG::2014-09-25 17:06:33,352::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`eb5d29f7-a8a1-480c-b0d7-c7dcd55dcd2a`::Granted request >Thread-69582::DEBUG::2014-09-25 17:06:33,352::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`07cb7b8c-4e8a-4c8b-b07b-3e304e560de3`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69582::DEBUG::2014-09-25 17:06:33,353::task::993::Storage.TaskManager.Task::(_decref) Task=`07cb7b8c-4e8a-4c8b-b07b-3e304e560de3`::ref 1 aborting False >Thread-69582::INFO::2014-09-25 17:06:33,357::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69582::DEBUG::2014-09-25 17:06:33,357::task::1191::Storage.TaskManager.Task::(prepare) Task=`07cb7b8c-4e8a-4c8b-b07b-3e304e560de3`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69582::DEBUG::2014-09-25 17:06:33,358::task::595::Storage.TaskManager.Task::(_updateState) Task=`07cb7b8c-4e8a-4c8b-b07b-3e304e560de3`::moving from state preparing -> state finished >Thread-69582::DEBUG::2014-09-25 17:06:33,358::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69582::DEBUG::2014-09-25 17:06:33,358::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69582::DEBUG::2014-09-25 17:06:33,358::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69582::DEBUG::2014-09-25 17:06:33,359::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69582::DEBUG::2014-09-25 17:06:33,359::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69582::DEBUG::2014-09-25 17:06:33,359::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69582::DEBUG::2014-09-25 17:06:33,359::task::993::Storage.TaskManager.Task::(_decref) Task=`07cb7b8c-4e8a-4c8b-b07b-3e304e560de3`::ref 0 aborting False >Thread-69582::DEBUG::2014-09-25 17:06:33,360::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69582::DEBUG::2014-09-25 17:06:33,360::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:33,830::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:33,832::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69583::DEBUG::2014-09-25 17:06:33,833::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:33,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:33,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212127 s, 48.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:35,320::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:35,320::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:35,321::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:35,321::__init__::232::IOProcess::(_processLogs) (107020) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:35,324::__init__::232::IOProcess::(_processLogs) (107020) Queuing response >Thread-81::DEBUG::2014-09-25 17:06:35,325::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:06:35,339::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00346382 s, 107 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:35,340::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:35,340::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:35,341::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:35,341::__init__::232::IOProcess::(_processLogs) (107021) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:35,343::__init__::232::IOProcess::(_processLogs) (107021) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:35,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:35,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168416 s, 60.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:36,846::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:36,848::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69584::DEBUG::2014-09-25 17:06:36,850::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:37,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:37,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211646 s, 48.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:39,664::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:39,664::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:39,665::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:39,665::__init__::232::IOProcess::(_processLogs) (107022) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:39,682::__init__::232::IOProcess::(_processLogs) (107022) Queuing response >Thread-79::DEBUG::2014-09-25 17:06:39,683::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:06:39,699::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00302856 s, 186 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:39,700::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:39,700::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:39,701::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:39,701::__init__::232::IOProcess::(_processLogs) (107023) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:39,707::__init__::232::IOProcess::(_processLogs) (107023) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:39,708::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:39,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:39,708::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:39,709::__init__::232::IOProcess::(_processLogs) (107024) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:39,712::__init__::232::IOProcess::(_processLogs) (107024) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:39,713::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:39,713::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:39,713::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:39,714::__init__::232::IOProcess::(_processLogs) (107025) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:39,719::__init__::232::IOProcess::(_processLogs) (107025) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:39,719::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:39,720::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:39,720::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:39,720::__init__::232::IOProcess::(_processLogs) (107026) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:39,735::__init__::232::IOProcess::(_processLogs) (107026) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:39,862::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:39,863::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69585::DEBUG::2014-09-25 17:06:39,865::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:40,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:40,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0126726 s, 80.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:41,777::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:41,777::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:41,777::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:41,778::__init__::232::IOProcess::(_processLogs) (107027) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:41,778::__init__::232::IOProcess::(_processLogs) (107027) Queuing response >Thread-80::DEBUG::2014-09-25 17:06:41,779::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:06:41,788::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000371806 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:41,790::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:41,790::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:41,791::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:41,792::__init__::232::IOProcess::(_processLogs) (107028) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:41,792::__init__::232::IOProcess::(_processLogs) (107028) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:42,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:42,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.262443 s, 3.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:42,874::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:42,876::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69586::DEBUG::2014-09-25 17:06:42,878::task::595::Storage.TaskManager.Task::(_updateState) Task=`7f79fc13-11f2-4be4-980e-6f272914e689`::moving from state init -> state preparing >Thread-69586::INFO::2014-09-25 17:06:42,879::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69586::INFO::2014-09-25 17:06:42,879::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00302856', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000371806', 'lastCheck': '1.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00346382', 'lastCheck': '7.5', 'valid': True}} >Thread-69586::DEBUG::2014-09-25 17:06:42,879::task::1191::Storage.TaskManager.Task::(prepare) Task=`7f79fc13-11f2-4be4-980e-6f272914e689`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00302856', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000371806', 'lastCheck': '1.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00346382', 'lastCheck': '7.5', 'valid': True}} >Thread-69586::DEBUG::2014-09-25 17:06:42,880::task::595::Storage.TaskManager.Task::(_updateState) Task=`7f79fc13-11f2-4be4-980e-6f272914e689`::moving from state preparing -> state finished >Thread-69586::DEBUG::2014-09-25 17:06:42,880::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69586::DEBUG::2014-09-25 17:06:42,880::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69586::DEBUG::2014-09-25 17:06:42,880::task::993::Storage.TaskManager.Task::(_decref) Task=`7f79fc13-11f2-4be4-980e-6f272914e689`::ref 0 aborting False >Thread-69586::DEBUG::2014-09-25 17:06:42,883::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:42,897::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:42,898::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69587::DEBUG::2014-09-25 17:06:42,900::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:43,383::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69588::DEBUG::2014-09-25 17:06:43,384::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:06:43,384::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69588::DEBUG::2014-09-25 17:06:43,386::task::595::Storage.TaskManager.Task::(_updateState) Task=`a2c79486-c21f-4841-9fc2-4efd448a639d`::moving from state init -> state preparing >Thread-69588::INFO::2014-09-25 17:06:43,387::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69588::INFO::2014-09-25 17:06:43,387::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69588::DEBUG::2014-09-25 17:06:43,387::task::1191::Storage.TaskManager.Task::(prepare) Task=`a2c79486-c21f-4841-9fc2-4efd448a639d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69588::DEBUG::2014-09-25 17:06:43,388::task::595::Storage.TaskManager.Task::(_updateState) Task=`a2c79486-c21f-4841-9fc2-4efd448a639d`::moving from state preparing -> state finished >Thread-69588::DEBUG::2014-09-25 17:06:43,388::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69588::DEBUG::2014-09-25 17:06:43,388::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69588::DEBUG::2014-09-25 17:06:43,388::task::993::Storage.TaskManager.Task::(_decref) Task=`a2c79486-c21f-4841-9fc2-4efd448a639d`::ref 0 aborting False >Thread-69588::DEBUG::2014-09-25 17:06:43,389::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69588::DEBUG::2014-09-25 17:06:43,389::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:43,399::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69589::DEBUG::2014-09-25 17:06:43,400::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:06:43,400::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69589::DEBUG::2014-09-25 17:06:43,402::task::595::Storage.TaskManager.Task::(_updateState) Task=`bd12ff53-990f-4cc7-8151-0ee58c28a25e`::moving from state init -> state preparing >Thread-69589::INFO::2014-09-25 17:06:43,402::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69589::DEBUG::2014-09-25 17:06:43,403::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`33a01531-59cb-44ca-a18a-04221b3903a5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69589::DEBUG::2014-09-25 17:06:43,403::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69589::DEBUG::2014-09-25 17:06:43,404::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69589::DEBUG::2014-09-25 17:06:43,404::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`33a01531-59cb-44ca-a18a-04221b3903a5`::Granted request >Thread-69589::DEBUG::2014-09-25 17:06:43,405::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`bd12ff53-990f-4cc7-8151-0ee58c28a25e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69589::DEBUG::2014-09-25 17:06:43,405::task::993::Storage.TaskManager.Task::(_decref) Task=`bd12ff53-990f-4cc7-8151-0ee58c28a25e`::ref 1 aborting False >Thread-69589::INFO::2014-09-25 17:06:43,409::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69589::DEBUG::2014-09-25 17:06:43,410::task::1191::Storage.TaskManager.Task::(prepare) Task=`bd12ff53-990f-4cc7-8151-0ee58c28a25e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69589::DEBUG::2014-09-25 17:06:43,410::task::595::Storage.TaskManager.Task::(_updateState) Task=`bd12ff53-990f-4cc7-8151-0ee58c28a25e`::moving from state preparing -> state finished >Thread-69589::DEBUG::2014-09-25 17:06:43,410::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69589::DEBUG::2014-09-25 17:06:43,410::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69589::DEBUG::2014-09-25 17:06:43,411::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69589::DEBUG::2014-09-25 17:06:43,411::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69589::DEBUG::2014-09-25 17:06:43,411::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69589::DEBUG::2014-09-25 17:06:43,411::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69589::DEBUG::2014-09-25 17:06:43,412::task::993::Storage.TaskManager.Task::(_decref) Task=`bd12ff53-990f-4cc7-8151-0ee58c28a25e`::ref 0 aborting False >Thread-69589::DEBUG::2014-09-25 17:06:43,412::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69589::DEBUG::2014-09-25 17:06:43,413::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:44,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:44,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0225718 s, 45.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:45,345::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:45,345::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:45,346::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:45,346::__init__::232::IOProcess::(_processLogs) (107029) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:45,346::__init__::232::IOProcess::(_processLogs) (107029) Queuing response >Thread-81::DEBUG::2014-09-25 17:06:45,348::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:06:45,356::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00036154 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:45,358::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:45,358::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:45,358::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:45,358::__init__::232::IOProcess::(_processLogs) (107030) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:45,359::__init__::232::IOProcess::(_processLogs) (107030) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:45,944::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:45,946::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69590::DEBUG::2014-09-25 17:06:45,948::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:46,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:46,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0240812 s, 42.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:06:48,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:48,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173196 s, 59.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:48,962::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:48,963::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69591::DEBUG::2014-09-25 17:06:48,965::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:06:49,737::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:49,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:49,738::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:49,738::__init__::232::IOProcess::(_processLogs) (107031) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:49,739::__init__::232::IOProcess::(_processLogs) (107031) Queuing response >Thread-79::DEBUG::2014-09-25 17:06:49,740::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:06:49,755::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00309053 s, 182 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:49,756::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:49,757::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:49,757::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:49,757::__init__::232::IOProcess::(_processLogs) (107032) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:49,759::__init__::232::IOProcess::(_processLogs) (107032) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:49,760::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:49,760::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:49,761::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:49,761::__init__::232::IOProcess::(_processLogs) (107033) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:49,761::__init__::232::IOProcess::(_processLogs) (107033) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:49,762::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:49,762::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:49,763::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:49,763::__init__::232::IOProcess::(_processLogs) (107034) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:49,763::__init__::232::IOProcess::(_processLogs) (107034) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:49,764::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:49,764::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:49,764::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:49,765::__init__::232::IOProcess::(_processLogs) (107035) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:49,765::__init__::232::IOProcess::(_processLogs) (107035) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:50,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:50,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145108 s, 70.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:51,793::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:51,793::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:51,793::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:51,794::__init__::232::IOProcess::(_processLogs) (107036) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:51,794::__init__::232::IOProcess::(_processLogs) (107036) Queuing response >Thread-80::DEBUG::2014-09-25 17:06:51,795::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:06:51,804::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000482467 s, 783 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:51,805::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:51,806::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:51,806::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:51,806::__init__::232::IOProcess::(_processLogs) (107037) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:51,807::__init__::232::IOProcess::(_processLogs) (107037) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:51,984::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:51,985::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69592::DEBUG::2014-09-25 17:06:51,987::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:52,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:52,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159999 s, 64.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:53,438::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69593::DEBUG::2014-09-25 17:06:53,439::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:06:53,439::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69593::DEBUG::2014-09-25 17:06:53,442::task::595::Storage.TaskManager.Task::(_updateState) Task=`9c11322b-307e-4b3f-8500-c19182dd6dc9`::moving from state init -> state preparing >Thread-69593::INFO::2014-09-25 17:06:53,442::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69593::INFO::2014-09-25 17:06:53,443::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69593::DEBUG::2014-09-25 17:06:53,443::task::1191::Storage.TaskManager.Task::(prepare) Task=`9c11322b-307e-4b3f-8500-c19182dd6dc9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69593::DEBUG::2014-09-25 17:06:53,443::task::595::Storage.TaskManager.Task::(_updateState) Task=`9c11322b-307e-4b3f-8500-c19182dd6dc9`::moving from state preparing -> state finished >Thread-69593::DEBUG::2014-09-25 17:06:53,443::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69593::DEBUG::2014-09-25 17:06:53,444::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69593::DEBUG::2014-09-25 17:06:53,444::task::993::Storage.TaskManager.Task::(_decref) Task=`9c11322b-307e-4b3f-8500-c19182dd6dc9`::ref 0 aborting False >Thread-69593::DEBUG::2014-09-25 17:06:53,444::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69593::DEBUG::2014-09-25 17:06:53,445::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:53,459::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69594::DEBUG::2014-09-25 17:06:53,460::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:06:53,460::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69594::DEBUG::2014-09-25 17:06:53,462::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e8ea468-dc18-4a57-933b-786076a597b8`::moving from state init -> state preparing >Thread-69594::INFO::2014-09-25 17:06:53,462::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69594::DEBUG::2014-09-25 17:06:53,463::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`14d57867-e27a-4462-8226-84d748806d06`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69594::DEBUG::2014-09-25 17:06:53,463::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69594::DEBUG::2014-09-25 17:06:53,464::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69594::DEBUG::2014-09-25 17:06:53,464::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`14d57867-e27a-4462-8226-84d748806d06`::Granted request >Thread-69594::DEBUG::2014-09-25 17:06:53,465::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5e8ea468-dc18-4a57-933b-786076a597b8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69594::DEBUG::2014-09-25 17:06:53,466::task::993::Storage.TaskManager.Task::(_decref) Task=`5e8ea468-dc18-4a57-933b-786076a597b8`::ref 1 aborting False >Thread-69594::INFO::2014-09-25 17:06:53,470::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69594::DEBUG::2014-09-25 17:06:53,470::task::1191::Storage.TaskManager.Task::(prepare) Task=`5e8ea468-dc18-4a57-933b-786076a597b8`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69594::DEBUG::2014-09-25 17:06:53,471::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e8ea468-dc18-4a57-933b-786076a597b8`::moving from state preparing -> state finished >Thread-69594::DEBUG::2014-09-25 17:06:53,471::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69594::DEBUG::2014-09-25 17:06:53,471::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69594::DEBUG::2014-09-25 17:06:53,471::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69594::DEBUG::2014-09-25 17:06:53,472::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69594::DEBUG::2014-09-25 17:06:53,472::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69594::DEBUG::2014-09-25 17:06:53,472::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69594::DEBUG::2014-09-25 17:06:53,473::task::993::Storage.TaskManager.Task::(_decref) Task=`5e8ea468-dc18-4a57-933b-786076a597b8`::ref 0 aborting False >Thread-69594::DEBUG::2014-09-25 17:06:53,473::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69594::DEBUG::2014-09-25 17:06:53,473::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:54,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:54,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199741 s, 51.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:55,006::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:55,008::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69595::DEBUG::2014-09-25 17:06:55,010::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:06:55,361::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:55,362::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:55,362::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:55,362::__init__::232::IOProcess::(_processLogs) (107038) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:55,373::__init__::232::IOProcess::(_processLogs) (107038) Queuing response >Thread-81::DEBUG::2014-09-25 17:06:55,373::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:06:55,396::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00469676 s, 79.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:55,397::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:55,397::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:55,398::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:55,398::__init__::232::IOProcess::(_processLogs) (107039) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:55,402::__init__::232::IOProcess::(_processLogs) (107039) Queuing response >Dummy-148::DEBUG::2014-09-25 17:06:56,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:56,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129029 s, 79.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:58,026::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:58,028::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69596::DEBUG::2014-09-25 17:06:58,030::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4b55765-3e75-4822-94cd-a0897add4883`::moving from state init -> state preparing >Thread-69596::INFO::2014-09-25 17:06:58,031::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69596::INFO::2014-09-25 17:06:58,031::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00309053', 'lastCheck': '8.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000482467', 'lastCheck': '6.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00469676', 'lastCheck': '2.6', 'valid': True}} >Thread-69596::DEBUG::2014-09-25 17:06:58,031::task::1191::Storage.TaskManager.Task::(prepare) Task=`c4b55765-3e75-4822-94cd-a0897add4883`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00309053', 'lastCheck': '8.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000482467', 'lastCheck': '6.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00469676', 'lastCheck': '2.6', 'valid': True}} >Thread-69596::DEBUG::2014-09-25 17:06:58,032::task::595::Storage.TaskManager.Task::(_updateState) Task=`c4b55765-3e75-4822-94cd-a0897add4883`::moving from state preparing -> state finished >Thread-69596::DEBUG::2014-09-25 17:06:58,032::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69596::DEBUG::2014-09-25 17:06:58,032::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69596::DEBUG::2014-09-25 17:06:58,032::task::993::Storage.TaskManager.Task::(_decref) Task=`c4b55765-3e75-4822-94cd-a0897add4883`::ref 0 aborting False >Thread-69596::DEBUG::2014-09-25 17:06:58,035::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:06:58,057::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:06:58,058::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69597::DEBUG::2014-09-25 17:06:58,060::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:06:58,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:06:58,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138224 s, 74.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:59,766::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:59,767::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:59,767::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:59,767::__init__::232::IOProcess::(_processLogs) (107040) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:59,768::__init__::232::IOProcess::(_processLogs) (107040) Queuing response >Thread-79::DEBUG::2014-09-25 17:06:59,769::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:06:59,778::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000442782 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:06:59,779::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:59,779::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:59,780::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:59,780::__init__::232::IOProcess::(_processLogs) (107041) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:06:59,780::__init__::232::IOProcess::(_processLogs) (107041) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:59,782::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:59,782::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:59,783::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:59,783::__init__::232::IOProcess::(_processLogs) (107042) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:59,783::__init__::232::IOProcess::(_processLogs) (107042) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:59,784::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:59,784::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:59,784::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:59,784::__init__::232::IOProcess::(_processLogs) (107043) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:59,785::__init__::232::IOProcess::(_processLogs) (107043) Queuing response >Thread-29::DEBUG::2014-09-25 17:06:59,785::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:06:59,786::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:06:59,786::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:06:59,786::__init__::232::IOProcess::(_processLogs) (107044) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:06:59,786::__init__::232::IOProcess::(_processLogs) (107044) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:00,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:00,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184822 s, 55.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:01,119::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:01,121::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69598::DEBUG::2014-09-25 17:07:01,123::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:07:01,809::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:01,809::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:01,810::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:01,810::__init__::232::IOProcess::(_processLogs) (107045) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:01,810::__init__::232::IOProcess::(_processLogs) (107045) Queuing response >Thread-80::DEBUG::2014-09-25 17:07:01,812::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:07:01,821::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000419494 s, 901 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:01,822::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:01,822::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:01,823::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:01,823::__init__::232::IOProcess::(_processLogs) (107046) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:01,825::__init__::232::IOProcess::(_processLogs) (107046) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:02,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:03,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161954 s, 63.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:03,528::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69599::DEBUG::2014-09-25 17:07:03,529::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:07:03,529::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69599::DEBUG::2014-09-25 17:07:03,531::task::595::Storage.TaskManager.Task::(_updateState) Task=`aafb2937-cd74-4639-ad8a-e9bd7b7351ad`::moving from state init -> state preparing >Thread-69599::INFO::2014-09-25 17:07:03,532::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69599::INFO::2014-09-25 17:07:03,532::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69599::DEBUG::2014-09-25 17:07:03,533::task::1191::Storage.TaskManager.Task::(prepare) Task=`aafb2937-cd74-4639-ad8a-e9bd7b7351ad`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69599::DEBUG::2014-09-25 17:07:03,533::task::595::Storage.TaskManager.Task::(_updateState) Task=`aafb2937-cd74-4639-ad8a-e9bd7b7351ad`::moving from state preparing -> state finished >Thread-69599::DEBUG::2014-09-25 17:07:03,533::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69599::DEBUG::2014-09-25 17:07:03,533::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69599::DEBUG::2014-09-25 17:07:03,533::task::993::Storage.TaskManager.Task::(_decref) Task=`aafb2937-cd74-4639-ad8a-e9bd7b7351ad`::ref 0 aborting False >Thread-69599::DEBUG::2014-09-25 17:07:03,534::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69599::DEBUG::2014-09-25 17:07:03,534::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:03,548::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:03,549::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69600::DEBUG::2014-09-25 17:07:03,549::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69600::DEBUG::2014-09-25 17:07:03,551::task::595::Storage.TaskManager.Task::(_updateState) Task=`09d6537d-a545-49a3-8cbc-53f8d8cf39cc`::moving from state init -> state preparing >Thread-69600::INFO::2014-09-25 17:07:03,551::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69600::DEBUG::2014-09-25 17:07:03,552::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`bd2cf01d-1a4a-46cd-b93a-35d8e68ea8d4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69600::DEBUG::2014-09-25 17:07:03,552::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69600::DEBUG::2014-09-25 17:07:03,552::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69600::DEBUG::2014-09-25 17:07:03,553::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`bd2cf01d-1a4a-46cd-b93a-35d8e68ea8d4`::Granted request >Thread-69600::DEBUG::2014-09-25 17:07:03,553::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`09d6537d-a545-49a3-8cbc-53f8d8cf39cc`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69600::DEBUG::2014-09-25 17:07:03,553::task::993::Storage.TaskManager.Task::(_decref) Task=`09d6537d-a545-49a3-8cbc-53f8d8cf39cc`::ref 1 aborting False >Thread-69600::INFO::2014-09-25 17:07:03,558::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69600::DEBUG::2014-09-25 17:07:03,558::task::1191::Storage.TaskManager.Task::(prepare) Task=`09d6537d-a545-49a3-8cbc-53f8d8cf39cc`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69600::DEBUG::2014-09-25 17:07:03,558::task::595::Storage.TaskManager.Task::(_updateState) Task=`09d6537d-a545-49a3-8cbc-53f8d8cf39cc`::moving from state preparing -> state finished >Thread-69600::DEBUG::2014-09-25 17:07:03,559::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69600::DEBUG::2014-09-25 17:07:03,559::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69600::DEBUG::2014-09-25 17:07:03,559::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69600::DEBUG::2014-09-25 17:07:03,559::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69600::DEBUG::2014-09-25 17:07:03,560::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69600::DEBUG::2014-09-25 17:07:03,560::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69600::DEBUG::2014-09-25 17:07:03,560::task::993::Storage.TaskManager.Task::(_decref) Task=`09d6537d-a545-49a3-8cbc-53f8d8cf39cc`::ref 0 aborting False >Thread-69600::DEBUG::2014-09-25 17:07:03,561::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69600::DEBUG::2014-09-25 17:07:03,561::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:04,140::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:04,141::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69601::DEBUG::2014-09-25 17:07:04,143::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:05,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:05,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0196711 s, 52.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:05,404::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:05,405::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:05,405::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:05,405::__init__::232::IOProcess::(_processLogs) (107047) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:05,407::__init__::232::IOProcess::(_processLogs) (107047) Queuing response >Thread-81::DEBUG::2014-09-25 17:07:05,409::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:07:05,422::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00380856 s, 97.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:05,423::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:05,424::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:05,424::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:05,424::__init__::232::IOProcess::(_processLogs) (107048) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:05,425::__init__::232::IOProcess::(_processLogs) (107048) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:07,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:07,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148185 s, 69.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:07,158::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:07,160::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69602::DEBUG::2014-09-25 17:07:07,162::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:09,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:09,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.213833 s, 4.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:09,788::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:09,788::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:09,789::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:09,789::__init__::232::IOProcess::(_processLogs) (107049) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:09,793::__init__::232::IOProcess::(_processLogs) (107049) Queuing response >Thread-79::DEBUG::2014-09-25 17:07:09,794::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:07:09,808::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00394223 s, 143 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:09,809::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:09,809::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:09,810::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:09,810::__init__::232::IOProcess::(_processLogs) (107050) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:09,813::__init__::232::IOProcess::(_processLogs) (107050) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:09,814::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:09,815::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:09,815::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:09,816::__init__::232::IOProcess::(_processLogs) (107051) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:09,816::__init__::232::IOProcess::(_processLogs) (107051) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:09,816::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:09,817::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:09,817::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:09,818::__init__::232::IOProcess::(_processLogs) (107052) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:09,818::__init__::232::IOProcess::(_processLogs) (107052) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:09,818::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:09,819::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:09,820::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:09,820::__init__::232::IOProcess::(_processLogs) (107053) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:09,820::__init__::232::IOProcess::(_processLogs) (107053) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:10,178::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:10,179::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69603::DEBUG::2014-09-25 17:07:10,181::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:11,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:11,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0221335 s, 46.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:11,826::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:11,826::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:11,826::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:11,826::__init__::232::IOProcess::(_processLogs) (107054) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:11,827::__init__::232::IOProcess::(_processLogs) (107054) Queuing response >Thread-80::DEBUG::2014-09-25 17:07:11,828::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:07:11,837::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000468968 s, 806 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:11,838::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:11,838::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:11,839::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:11,840::__init__::232::IOProcess::(_processLogs) (107055) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:11,841::__init__::232::IOProcess::(_processLogs) (107055) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:13,197::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:13,198::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69604::DEBUG::2014-09-25 17:07:13,200::task::595::Storage.TaskManager.Task::(_updateState) Task=`815d30ba-f5d2-4c7c-8b9e-4ad846adc346`::moving from state init -> state preparing >Thread-69604::INFO::2014-09-25 17:07:13,201::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69604::INFO::2014-09-25 17:07:13,201::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00394223', 'lastCheck': '3.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000468968', 'lastCheck': '1.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00380856', 'lastCheck': '7.8', 'valid': True}} >Thread-69604::DEBUG::2014-09-25 17:07:13,201::task::1191::Storage.TaskManager.Task::(prepare) Task=`815d30ba-f5d2-4c7c-8b9e-4ad846adc346`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00394223', 'lastCheck': '3.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000468968', 'lastCheck': '1.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00380856', 'lastCheck': '7.8', 'valid': True}} >Thread-69604::DEBUG::2014-09-25 17:07:13,202::task::595::Storage.TaskManager.Task::(_updateState) Task=`815d30ba-f5d2-4c7c-8b9e-4ad846adc346`::moving from state preparing -> state finished >Thread-69604::DEBUG::2014-09-25 17:07:13,202::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69604::DEBUG::2014-09-25 17:07:13,202::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69604::DEBUG::2014-09-25 17:07:13,202::task::993::Storage.TaskManager.Task::(_decref) Task=`815d30ba-f5d2-4c7c-8b9e-4ad846adc346`::ref 0 aborting False >Thread-69604::DEBUG::2014-09-25 17:07:13,205::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:13,225::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:13,226::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69605::DEBUG::2014-09-25 17:07:13,228::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:13,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:13,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149365 s, 68.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:13,590::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:13,592::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69606::DEBUG::2014-09-25 17:07:13,592::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69606::DEBUG::2014-09-25 17:07:13,594::task::595::Storage.TaskManager.Task::(_updateState) Task=`0144186c-c8e5-4c48-835a-e58c8952816e`::moving from state init -> state preparing >Thread-69606::INFO::2014-09-25 17:07:13,594::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69606::INFO::2014-09-25 17:07:13,595::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69606::DEBUG::2014-09-25 17:07:13,595::task::1191::Storage.TaskManager.Task::(prepare) Task=`0144186c-c8e5-4c48-835a-e58c8952816e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69606::DEBUG::2014-09-25 17:07:13,595::task::595::Storage.TaskManager.Task::(_updateState) Task=`0144186c-c8e5-4c48-835a-e58c8952816e`::moving from state preparing -> state finished >Thread-69606::DEBUG::2014-09-25 17:07:13,595::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69606::DEBUG::2014-09-25 17:07:13,596::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69606::DEBUG::2014-09-25 17:07:13,596::task::993::Storage.TaskManager.Task::(_decref) Task=`0144186c-c8e5-4c48-835a-e58c8952816e`::ref 0 aborting False >Thread-69606::DEBUG::2014-09-25 17:07:13,596::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69606::DEBUG::2014-09-25 17:07:13,597::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:13,610::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69607::DEBUG::2014-09-25 17:07:13,611::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:07:13,611::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69607::DEBUG::2014-09-25 17:07:13,613::task::595::Storage.TaskManager.Task::(_updateState) Task=`b8b95ada-2946-4bd2-8857-538f4ef66e3c`::moving from state init -> state preparing >Thread-69607::INFO::2014-09-25 17:07:13,614::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69607::DEBUG::2014-09-25 17:07:13,614::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`56649cf8-a53c-433a-861d-71b3f6254695`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69607::DEBUG::2014-09-25 17:07:13,614::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69607::DEBUG::2014-09-25 17:07:13,615::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69607::DEBUG::2014-09-25 17:07:13,615::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`56649cf8-a53c-433a-861d-71b3f6254695`::Granted request >Thread-69607::DEBUG::2014-09-25 17:07:13,616::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b8b95ada-2946-4bd2-8857-538f4ef66e3c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69607::DEBUG::2014-09-25 17:07:13,616::task::993::Storage.TaskManager.Task::(_decref) Task=`b8b95ada-2946-4bd2-8857-538f4ef66e3c`::ref 1 aborting False >Thread-69607::INFO::2014-09-25 17:07:13,620::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69607::DEBUG::2014-09-25 17:07:13,621::task::1191::Storage.TaskManager.Task::(prepare) Task=`b8b95ada-2946-4bd2-8857-538f4ef66e3c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69607::DEBUG::2014-09-25 17:07:13,621::task::595::Storage.TaskManager.Task::(_updateState) Task=`b8b95ada-2946-4bd2-8857-538f4ef66e3c`::moving from state preparing -> state finished >Thread-69607::DEBUG::2014-09-25 17:07:13,621::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69607::DEBUG::2014-09-25 17:07:13,621::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69607::DEBUG::2014-09-25 17:07:13,622::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69607::DEBUG::2014-09-25 17:07:13,622::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69607::DEBUG::2014-09-25 17:07:13,622::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69607::DEBUG::2014-09-25 17:07:13,622::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69607::DEBUG::2014-09-25 17:07:13,623::task::993::Storage.TaskManager.Task::(_decref) Task=`b8b95ada-2946-4bd2-8857-538f4ef66e3c`::ref 0 aborting False >Thread-69607::DEBUG::2014-09-25 17:07:13,623::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69607::DEBUG::2014-09-25 17:07:13,624::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:07:15,427::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:15,428::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:15,428::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:15,428::__init__::232::IOProcess::(_processLogs) (107056) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:15,434::__init__::232::IOProcess::(_processLogs) (107056) Queuing response >Thread-81::DEBUG::2014-09-25 17:07:15,435::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:07:15,455::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00423758 s, 87.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:15,456::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:15,456::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:15,457::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:15,457::__init__::232::IOProcess::(_processLogs) (107057) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:15,459::__init__::232::IOProcess::(_processLogs) (107057) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:15,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:15,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211218 s, 48.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:16,291::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:16,294::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69608::DEBUG::2014-09-25 17:07:16,300::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:17,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:17,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015172 s, 67.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:19,317::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:19,318::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69609::DEBUG::2014-09-25 17:07:19,320::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:19,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:19,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016078 s, 63.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:19,821::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:19,821::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:19,821::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:19,822::__init__::232::IOProcess::(_processLogs) (107058) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:19,828::__init__::232::IOProcess::(_processLogs) (107058) Queuing response >Thread-79::DEBUG::2014-09-25 17:07:19,829::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:07:19,845::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00728516 s, 77.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:19,847::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:19,847::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:19,847::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:19,847::__init__::232::IOProcess::(_processLogs) (107059) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:19,855::__init__::232::IOProcess::(_processLogs) (107059) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:19,856::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:19,856::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:19,857::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:19,857::__init__::232::IOProcess::(_processLogs) (107060) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:19,858::__init__::232::IOProcess::(_processLogs) (107060) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:19,858::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:19,859::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:19,859::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:19,860::__init__::232::IOProcess::(_processLogs) (107061) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:19,860::__init__::232::IOProcess::(_processLogs) (107061) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:19,860::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:19,861::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:19,862::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:19,862::__init__::232::IOProcess::(_processLogs) (107062) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:19,862::__init__::232::IOProcess::(_processLogs) (107062) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:21,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:21,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135864 s, 75.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:21,841::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:21,842::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:21,842::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:21,842::__init__::232::IOProcess::(_processLogs) (107063) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:21,843::__init__::232::IOProcess::(_processLogs) (107063) Queuing response >Thread-80::DEBUG::2014-09-25 17:07:21,844::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:07:21,854::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000845994 s, 447 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:21,856::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:21,856::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:21,857::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:21,858::__init__::232::IOProcess::(_processLogs) (107064) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:21,858::__init__::232::IOProcess::(_processLogs) (107064) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:22,338::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:22,339::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69610::DEBUG::2014-09-25 17:07:22,341::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:23,652::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69611::DEBUG::2014-09-25 17:07:23,653::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:07:23,653::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69611::DEBUG::2014-09-25 17:07:23,655::task::595::Storage.TaskManager.Task::(_updateState) Task=`76a44620-5e98-4eef-b636-a3dd2a2e60c2`::moving from state init -> state preparing >Thread-69611::INFO::2014-09-25 17:07:23,656::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69611::INFO::2014-09-25 17:07:23,656::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69611::DEBUG::2014-09-25 17:07:23,656::task::1191::Storage.TaskManager.Task::(prepare) Task=`76a44620-5e98-4eef-b636-a3dd2a2e60c2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69611::DEBUG::2014-09-25 17:07:23,657::task::595::Storage.TaskManager.Task::(_updateState) Task=`76a44620-5e98-4eef-b636-a3dd2a2e60c2`::moving from state preparing -> state finished >Thread-69611::DEBUG::2014-09-25 17:07:23,657::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69611::DEBUG::2014-09-25 17:07:23,657::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69611::DEBUG::2014-09-25 17:07:23,657::task::993::Storage.TaskManager.Task::(_decref) Task=`76a44620-5e98-4eef-b636-a3dd2a2e60c2`::ref 0 aborting False >Thread-69611::DEBUG::2014-09-25 17:07:23,658::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69611::DEBUG::2014-09-25 17:07:23,658::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:23,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:23,674::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:23,676::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69612::DEBUG::2014-09-25 17:07:23,677::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69612::DEBUG::2014-09-25 17:07:23,680::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d90a6be-7edf-46a0-a5d0-fb066d2955e0`::moving from state init -> state preparing >Thread-69612::INFO::2014-09-25 17:07:23,680::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69612::DEBUG::2014-09-25 17:07:23,682::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4324d1b3-2910-47f6-a011-c7ef84c0c80a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69612::DEBUG::2014-09-25 17:07:23,682::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69612::DEBUG::2014-09-25 17:07:23,683::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69612::DEBUG::2014-09-25 17:07:23,683::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4324d1b3-2910-47f6-a011-c7ef84c0c80a`::Granted request >Thread-69612::DEBUG::2014-09-25 17:07:23,684::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`6d90a6be-7edf-46a0-a5d0-fb066d2955e0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69612::DEBUG::2014-09-25 17:07:23,684::task::993::Storage.TaskManager.Task::(_decref) Task=`6d90a6be-7edf-46a0-a5d0-fb066d2955e0`::ref 1 aborting False >Thread-69612::INFO::2014-09-25 17:07:23,703::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Dummy-148::DEBUG::2014-09-25 17:07:23,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0277945 s, 36.8 MB/s\n'; <rc> = 0 >Thread-69612::DEBUG::2014-09-25 17:07:23,705::task::1191::Storage.TaskManager.Task::(prepare) Task=`6d90a6be-7edf-46a0-a5d0-fb066d2955e0`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69612::DEBUG::2014-09-25 17:07:23,706::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d90a6be-7edf-46a0-a5d0-fb066d2955e0`::moving from state preparing -> state finished >Thread-69612::DEBUG::2014-09-25 17:07:23,708::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69612::DEBUG::2014-09-25 17:07:23,709::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69612::DEBUG::2014-09-25 17:07:23,711::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69612::DEBUG::2014-09-25 17:07:23,712::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69612::DEBUG::2014-09-25 17:07:23,713::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69612::DEBUG::2014-09-25 17:07:23,714::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69612::DEBUG::2014-09-25 17:07:23,716::task::993::Storage.TaskManager.Task::(_decref) Task=`6d90a6be-7edf-46a0-a5d0-fb066d2955e0`::ref 0 aborting False >Thread-69612::DEBUG::2014-09-25 17:07:23,718::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69612::DEBUG::2014-09-25 17:07:23,720::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:25,359::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:25,360::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69613::DEBUG::2014-09-25 17:07:25,362::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 17:07:25,461::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:07:25,465::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:25,466::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:25,466::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:25,466::__init__::232::IOProcess::(_processLogs) (107065) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:07:25,467::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:25,467::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:25,467::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:25,467::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:25,468::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:25,468::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:25,468::__init__::232::IOProcess::(_processLogs) (107066) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:07:25,468::__init__::232::IOProcess::(_processLogs) (107067) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:07:25,469::__init__::232::IOProcess::(_processLogs) (107066) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:25,470::__init__::232::IOProcess::(_processLogs) (107065) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:25,471::__init__::232::IOProcess::(_processLogs) (107067) Queuing response >Thread-81::DEBUG::2014-09-25 17:07:25,476::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:07:25,477::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:25,477::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:25,478::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:25,478::__init__::232::IOProcess::(_processLogs) (107068) Got request for method 'touch' >Thread-81::DEBUG::2014-09-25 17:07:25,522::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:07:25,522::__init__::232::IOProcess::(_processLogs) (107068) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:25,524::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:25,524::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:25,524::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:25,524::__init__::232::IOProcess::(_processLogs) (107069) Got request for method 'readfile' >Thread-81::DEBUG::2014-09-25 17:07:25,533::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:07:25,533::__init__::232::IOProcess::(_processLogs) (107069) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:25,535::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:25,536::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:25,536::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:25,537::__init__::232::IOProcess::(_processLogs) (107070) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:25,537::__init__::232::IOProcess::(_processLogs) (107070) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:25,537::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:25,538::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:25,538::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:25,538::__init__::232::IOProcess::(_processLogs) (107071) Got request for method 'glob' >Thread-81::DEBUG::2014-09-25 17:07:25,547::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:07:25,547::__init__::232::IOProcess::(_processLogs) (107071) Queuing response >Thread-81::INFO::2014-09-25 17:07:25,548::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:07:25,549::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:07:25,550::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:25,550::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:25,550::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:25,550::__init__::232::IOProcess::(_processLogs) (107072) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:25,552::__init__::232::IOProcess::(_processLogs) (107072) Queuing response >Thread-81::DEBUG::2014-09-25 17:07:25,553::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:07:25,567::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00461849 s, 80.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:25,568::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:25,569::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:25,569::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:25,569::__init__::232::IOProcess::(_processLogs) (107073) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:25,572::__init__::232::IOProcess::(_processLogs) (107073) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:25,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:25,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192974 s, 53.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:07:27,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:27,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150618 s, 68.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:28,377::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:28,378::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69617::DEBUG::2014-09-25 17:07:28,381::task::595::Storage.TaskManager.Task::(_updateState) Task=`23216c28-30a3-4e0e-943b-2cdf77315f45`::moving from state init -> state preparing >Thread-69617::INFO::2014-09-25 17:07:28,381::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69617::INFO::2014-09-25 17:07:28,382::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00728516', 'lastCheck': '8.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000845994', 'lastCheck': '6.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00461849', 'lastCheck': '2.8', 'valid': True}} >Thread-69617::DEBUG::2014-09-25 17:07:28,382::task::1191::Storage.TaskManager.Task::(prepare) Task=`23216c28-30a3-4e0e-943b-2cdf77315f45`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00728516', 'lastCheck': '8.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000845994', 'lastCheck': '6.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00461849', 'lastCheck': '2.8', 'valid': True}} >Thread-69617::DEBUG::2014-09-25 17:07:28,382::task::595::Storage.TaskManager.Task::(_updateState) Task=`23216c28-30a3-4e0e-943b-2cdf77315f45`::moving from state preparing -> state finished >Thread-69617::DEBUG::2014-09-25 17:07:28,382::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69617::DEBUG::2014-09-25 17:07:28,383::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69617::DEBUG::2014-09-25 17:07:28,383::task::993::Storage.TaskManager.Task::(_decref) Task=`23216c28-30a3-4e0e-943b-2cdf77315f45`::ref 0 aborting False >Thread-69617::DEBUG::2014-09-25 17:07:28,385::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:28,407::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:28,408::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69618::DEBUG::2014-09-25 17:07:28,409::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:29,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:29,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133134 s, 76.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:29,864::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:29,865::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:29,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:29,867::__init__::232::IOProcess::(_processLogs) (107074) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:29,868::__init__::232::IOProcess::(_processLogs) (107074) Queuing response >Thread-79::DEBUG::2014-09-25 17:07:29,870::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:07:29,885::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00348765 s, 161 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:29,888::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:29,889::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:29,889::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:29,890::__init__::232::IOProcess::(_processLogs) (107075) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:29,893::__init__::232::IOProcess::(_processLogs) (107075) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:29,895::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:29,896::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:29,898::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:29,899::__init__::232::IOProcess::(_processLogs) (107076) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:29,900::__init__::232::IOProcess::(_processLogs) (107076) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:29,901::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:29,902::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:29,904::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:29,905::__init__::232::IOProcess::(_processLogs) (107077) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:29,906::__init__::232::IOProcess::(_processLogs) (107077) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:29,907::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:29,908::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:29,909::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:29,910::__init__::232::IOProcess::(_processLogs) (107078) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:29,911::__init__::232::IOProcess::(_processLogs) (107078) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:31,476::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:31,478::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69619::DEBUG::2014-09-25 17:07:31,479::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:07:31,859::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:31,859::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:31,859::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:31,859::__init__::232::IOProcess::(_processLogs) (107079) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:31,860::__init__::232::IOProcess::(_processLogs) (107079) Queuing response >Thread-80::DEBUG::2014-09-25 17:07:31,861::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:07:31,870::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000349276 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:31,871::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:31,871::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:31,872::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:31,873::__init__::232::IOProcess::(_processLogs) (107080) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:31,873::__init__::232::IOProcess::(_processLogs) (107080) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:31,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:31,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132588 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:33,753::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:33,754::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69620::DEBUG::2014-09-25 17:07:33,754::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69620::DEBUG::2014-09-25 17:07:33,756::task::595::Storage.TaskManager.Task::(_updateState) Task=`523b31e8-303e-4533-acd3-58e0369306bc`::moving from state init -> state preparing >Thread-69620::INFO::2014-09-25 17:07:33,757::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69620::INFO::2014-09-25 17:07:33,757::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69620::DEBUG::2014-09-25 17:07:33,757::task::1191::Storage.TaskManager.Task::(prepare) Task=`523b31e8-303e-4533-acd3-58e0369306bc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69620::DEBUG::2014-09-25 17:07:33,758::task::595::Storage.TaskManager.Task::(_updateState) Task=`523b31e8-303e-4533-acd3-58e0369306bc`::moving from state preparing -> state finished >Thread-69620::DEBUG::2014-09-25 17:07:33,758::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69620::DEBUG::2014-09-25 17:07:33,758::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69620::DEBUG::2014-09-25 17:07:33,758::task::993::Storage.TaskManager.Task::(_decref) Task=`523b31e8-303e-4533-acd3-58e0369306bc`::ref 0 aborting False >Thread-69620::DEBUG::2014-09-25 17:07:33,759::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69620::DEBUG::2014-09-25 17:07:33,759::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:33,773::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:33,774::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69621::DEBUG::2014-09-25 17:07:33,774::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69621::DEBUG::2014-09-25 17:07:33,776::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc5db127-0317-4d14-b1a6-632ea00fa978`::moving from state init -> state preparing >Thread-69621::INFO::2014-09-25 17:07:33,776::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69621::DEBUG::2014-09-25 17:07:33,777::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`afafd743-0226-4b60-aff0-b3c139832812`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69621::DEBUG::2014-09-25 17:07:33,777::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69621::DEBUG::2014-09-25 17:07:33,779::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69621::DEBUG::2014-09-25 17:07:33,779::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`afafd743-0226-4b60-aff0-b3c139832812`::Granted request >Thread-69621::DEBUG::2014-09-25 17:07:33,779::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`fc5db127-0317-4d14-b1a6-632ea00fa978`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69621::DEBUG::2014-09-25 17:07:33,780::task::993::Storage.TaskManager.Task::(_decref) Task=`fc5db127-0317-4d14-b1a6-632ea00fa978`::ref 1 aborting False >Thread-69621::INFO::2014-09-25 17:07:33,784::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69621::DEBUG::2014-09-25 17:07:33,784::task::1191::Storage.TaskManager.Task::(prepare) Task=`fc5db127-0317-4d14-b1a6-632ea00fa978`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69621::DEBUG::2014-09-25 17:07:33,785::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc5db127-0317-4d14-b1a6-632ea00fa978`::moving from state preparing -> state finished >Thread-69621::DEBUG::2014-09-25 17:07:33,785::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69621::DEBUG::2014-09-25 17:07:33,785::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69621::DEBUG::2014-09-25 17:07:33,785::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69621::DEBUG::2014-09-25 17:07:33,786::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69621::DEBUG::2014-09-25 17:07:33,786::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69621::DEBUG::2014-09-25 17:07:33,786::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69621::DEBUG::2014-09-25 17:07:33,787::task::993::Storage.TaskManager.Task::(_decref) Task=`fc5db127-0317-4d14-b1a6-632ea00fa978`::ref 0 aborting False >Thread-69621::DEBUG::2014-09-25 17:07:33,787::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69621::DEBUG::2014-09-25 17:07:33,787::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:33,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:33,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135103 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:34,495::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:34,497::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69622::DEBUG::2014-09-25 17:07:34,499::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:07:35,574::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:35,574::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:35,575::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:35,575::__init__::232::IOProcess::(_processLogs) (107081) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:35,576::__init__::232::IOProcess::(_processLogs) (107081) Queuing response >Thread-81::DEBUG::2014-09-25 17:07:35,578::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:07:35,591::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00286245 s, 130 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:35,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:35,592::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:35,592::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:35,593::__init__::232::IOProcess::(_processLogs) (107082) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:35,594::__init__::232::IOProcess::(_processLogs) (107082) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:36,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:36,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192569 s, 53.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:37,514::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:37,515::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69623::DEBUG::2014-09-25 17:07:37,517::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:38,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:38,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175028 s, 58.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:39,915::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:39,915::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:39,915::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:39,916::__init__::232::IOProcess::(_processLogs) (107083) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:39,918::__init__::232::IOProcess::(_processLogs) (107083) Queuing response >Thread-79::DEBUG::2014-09-25 17:07:39,919::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:07:39,931::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00280905 s, 200 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:39,932::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:39,933::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:39,933::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:39,933::__init__::232::IOProcess::(_processLogs) (107084) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:39,935::__init__::232::IOProcess::(_processLogs) (107084) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:39,936::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:39,937::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:39,937::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:39,937::__init__::232::IOProcess::(_processLogs) (107085) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:39,939::__init__::232::IOProcess::(_processLogs) (107085) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:39,940::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:39,940::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:39,940::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:39,940::__init__::232::IOProcess::(_processLogs) (107086) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:39,942::__init__::232::IOProcess::(_processLogs) (107086) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:39,943::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:39,943::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:39,943::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:39,944::__init__::232::IOProcess::(_processLogs) (107087) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:39,948::__init__::232::IOProcess::(_processLogs) (107087) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:40,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:40,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168759 s, 60.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:40,531::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:40,532::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69624::DEBUG::2014-09-25 17:07:40,535::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:07:41,874::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:41,875::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:41,875::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:41,875::__init__::232::IOProcess::(_processLogs) (107088) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:41,876::__init__::232::IOProcess::(_processLogs) (107088) Queuing response >Thread-80::DEBUG::2014-09-25 17:07:41,877::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:07:41,887::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000516066 s, 732 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:41,888::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:41,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:41,888::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:41,888::__init__::232::IOProcess::(_processLogs) (107089) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:41,889::__init__::232::IOProcess::(_processLogs) (107089) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:42,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:42,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159592 s, 64.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:43,549::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:43,551::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69625::DEBUG::2014-09-25 17:07:43,553::task::595::Storage.TaskManager.Task::(_updateState) Task=`3d751130-ff2d-49ac-b33f-836b88818cd0`::moving from state init -> state preparing >Thread-69625::INFO::2014-09-25 17:07:43,554::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69625::INFO::2014-09-25 17:07:43,554::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00280905', 'lastCheck': '3.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000516066', 'lastCheck': '1.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00286245', 'lastCheck': '8.0', 'valid': True}} >Thread-69625::DEBUG::2014-09-25 17:07:43,555::task::1191::Storage.TaskManager.Task::(prepare) Task=`3d751130-ff2d-49ac-b33f-836b88818cd0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00280905', 'lastCheck': '3.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000516066', 'lastCheck': '1.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00286245', 'lastCheck': '8.0', 'valid': True}} >Thread-69625::DEBUG::2014-09-25 17:07:43,555::task::595::Storage.TaskManager.Task::(_updateState) Task=`3d751130-ff2d-49ac-b33f-836b88818cd0`::moving from state preparing -> state finished >Thread-69625::DEBUG::2014-09-25 17:07:43,555::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69625::DEBUG::2014-09-25 17:07:43,555::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69625::DEBUG::2014-09-25 17:07:43,555::task::993::Storage.TaskManager.Task::(_decref) Task=`3d751130-ff2d-49ac-b33f-836b88818cd0`::ref 0 aborting False >Thread-69625::DEBUG::2014-09-25 17:07:43,558::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:43,578::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:43,579::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69626::DEBUG::2014-09-25 17:07:43,581::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:43,814::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69627::DEBUG::2014-09-25 17:07:43,815::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:07:43,815::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69627::DEBUG::2014-09-25 17:07:43,817::task::595::Storage.TaskManager.Task::(_updateState) Task=`5163c4b1-6afe-4f46-8875-4d5715d76a41`::moving from state init -> state preparing >Thread-69627::INFO::2014-09-25 17:07:43,817::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69627::INFO::2014-09-25 17:07:43,818::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69627::DEBUG::2014-09-25 17:07:43,818::task::1191::Storage.TaskManager.Task::(prepare) Task=`5163c4b1-6afe-4f46-8875-4d5715d76a41`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69627::DEBUG::2014-09-25 17:07:43,818::task::595::Storage.TaskManager.Task::(_updateState) Task=`5163c4b1-6afe-4f46-8875-4d5715d76a41`::moving from state preparing -> state finished >Thread-69627::DEBUG::2014-09-25 17:07:43,818::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69627::DEBUG::2014-09-25 17:07:43,819::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69627::DEBUG::2014-09-25 17:07:43,819::task::993::Storage.TaskManager.Task::(_decref) Task=`5163c4b1-6afe-4f46-8875-4d5715d76a41`::ref 0 aborting False >Thread-69627::DEBUG::2014-09-25 17:07:43,819::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69627::DEBUG::2014-09-25 17:07:43,819::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:43,833::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69628::DEBUG::2014-09-25 17:07:43,834::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:07:43,834::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69628::DEBUG::2014-09-25 17:07:43,836::task::595::Storage.TaskManager.Task::(_updateState) Task=`d74b30a5-3f20-41f0-aa85-c379a9c851a4`::moving from state init -> state preparing >Thread-69628::INFO::2014-09-25 17:07:43,836::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69628::DEBUG::2014-09-25 17:07:43,837::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4f95151b-42dd-4510-8896-0c47390eaa54`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69628::DEBUG::2014-09-25 17:07:43,837::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69628::DEBUG::2014-09-25 17:07:43,837::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69628::DEBUG::2014-09-25 17:07:43,838::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4f95151b-42dd-4510-8896-0c47390eaa54`::Granted request >Thread-69628::DEBUG::2014-09-25 17:07:43,838::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`d74b30a5-3f20-41f0-aa85-c379a9c851a4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69628::DEBUG::2014-09-25 17:07:43,838::task::993::Storage.TaskManager.Task::(_decref) Task=`d74b30a5-3f20-41f0-aa85-c379a9c851a4`::ref 1 aborting False >Thread-69628::INFO::2014-09-25 17:07:43,843::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69628::DEBUG::2014-09-25 17:07:43,843::task::1191::Storage.TaskManager.Task::(prepare) Task=`d74b30a5-3f20-41f0-aa85-c379a9c851a4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69628::DEBUG::2014-09-25 17:07:43,843::task::595::Storage.TaskManager.Task::(_updateState) Task=`d74b30a5-3f20-41f0-aa85-c379a9c851a4`::moving from state preparing -> state finished >Thread-69628::DEBUG::2014-09-25 17:07:43,844::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69628::DEBUG::2014-09-25 17:07:43,844::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69628::DEBUG::2014-09-25 17:07:43,844::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69628::DEBUG::2014-09-25 17:07:43,845::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69628::DEBUG::2014-09-25 17:07:43,845::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69628::DEBUG::2014-09-25 17:07:43,845::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69628::DEBUG::2014-09-25 17:07:43,845::task::993::Storage.TaskManager.Task::(_decref) Task=`d74b30a5-3f20-41f0-aa85-c379a9c851a4`::ref 0 aborting False >Thread-69628::DEBUG::2014-09-25 17:07:43,846::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69628::DEBUG::2014-09-25 17:07:43,846::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:44,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:44,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169379 s, 60.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:45,596::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:45,597::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:45,597::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:45,597::__init__::232::IOProcess::(_processLogs) (107090) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:45,601::__init__::232::IOProcess::(_processLogs) (107090) Queuing response >Thread-81::DEBUG::2014-09-25 17:07:45,602::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:07:45,616::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00389632 s, 95.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:45,617::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:45,617::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:45,618::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:45,618::__init__::232::IOProcess::(_processLogs) (107091) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:45,622::__init__::232::IOProcess::(_processLogs) (107091) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:46,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:46,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.280441 s, 3.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:46,645::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:46,650::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69629::DEBUG::2014-09-25 17:07:46,658::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:48,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:48,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167168 s, 61.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:49,675::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:49,676::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69630::DEBUG::2014-09-25 17:07:49,678::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:07:49,950::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:49,950::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:49,950::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:49,950::__init__::232::IOProcess::(_processLogs) (107092) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:49,951::__init__::232::IOProcess::(_processLogs) (107092) Queuing response >Thread-79::DEBUG::2014-09-25 17:07:49,953::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:07:49,961::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000456758 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:49,962::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:49,963::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:49,963::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:49,963::__init__::232::IOProcess::(_processLogs) (107093) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:49,964::__init__::232::IOProcess::(_processLogs) (107093) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:49,965::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:49,966::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:49,966::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:49,966::__init__::232::IOProcess::(_processLogs) (107094) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:49,966::__init__::232::IOProcess::(_processLogs) (107094) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:49,967::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:49,967::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:49,968::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:49,968::__init__::232::IOProcess::(_processLogs) (107095) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:49,968::__init__::232::IOProcess::(_processLogs) (107095) Queuing response >Thread-29::DEBUG::2014-09-25 17:07:49,969::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:49,969::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:49,970::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:49,971::__init__::232::IOProcess::(_processLogs) (107096) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:07:49,971::__init__::232::IOProcess::(_processLogs) (107096) Queuing response >Dummy-148::DEBUG::2014-09-25 17:07:50,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:50,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179172 s, 57.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:51,891::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:51,892::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:51,892::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:51,892::__init__::232::IOProcess::(_processLogs) (107097) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:51,893::__init__::232::IOProcess::(_processLogs) (107097) Queuing response >Thread-80::DEBUG::2014-09-25 17:07:51,894::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:07:51,903::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00067448 s, 560 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:51,905::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:51,905::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:51,905::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:51,905::__init__::232::IOProcess::(_processLogs) (107098) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:51,906::__init__::232::IOProcess::(_processLogs) (107098) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:52,693::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:52,694::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69631::DEBUG::2014-09-25 17:07:52,696::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:52,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:52,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151409 s, 67.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:53,875::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69632::DEBUG::2014-09-25 17:07:53,876::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:07:53,876::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69632::DEBUG::2014-09-25 17:07:53,879::task::595::Storage.TaskManager.Task::(_updateState) Task=`bc21d952-2dc3-404c-8d45-915c78c0ee6e`::moving from state init -> state preparing >Thread-69632::INFO::2014-09-25 17:07:53,879::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69632::INFO::2014-09-25 17:07:53,880::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69632::DEBUG::2014-09-25 17:07:53,880::task::1191::Storage.TaskManager.Task::(prepare) Task=`bc21d952-2dc3-404c-8d45-915c78c0ee6e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69632::DEBUG::2014-09-25 17:07:53,880::task::595::Storage.TaskManager.Task::(_updateState) Task=`bc21d952-2dc3-404c-8d45-915c78c0ee6e`::moving from state preparing -> state finished >Thread-69632::DEBUG::2014-09-25 17:07:53,880::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69632::DEBUG::2014-09-25 17:07:53,881::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69632::DEBUG::2014-09-25 17:07:53,881::task::993::Storage.TaskManager.Task::(_decref) Task=`bc21d952-2dc3-404c-8d45-915c78c0ee6e`::ref 0 aborting False >Thread-69632::DEBUG::2014-09-25 17:07:53,881::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69632::DEBUG::2014-09-25 17:07:53,882::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:53,901::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69633::DEBUG::2014-09-25 17:07:53,902::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:07:53,902::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69633::DEBUG::2014-09-25 17:07:53,904::task::595::Storage.TaskManager.Task::(_updateState) Task=`6ecbc125-0029-4737-9ac6-ebb563a0f22f`::moving from state init -> state preparing >Thread-69633::INFO::2014-09-25 17:07:53,905::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69633::DEBUG::2014-09-25 17:07:53,905::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2beb9e9d-0506-4a93-a499-69172bf6fa65`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69633::DEBUG::2014-09-25 17:07:53,906::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69633::DEBUG::2014-09-25 17:07:53,906::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69633::DEBUG::2014-09-25 17:07:53,906::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2beb9e9d-0506-4a93-a499-69172bf6fa65`::Granted request >Thread-69633::DEBUG::2014-09-25 17:07:53,907::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`6ecbc125-0029-4737-9ac6-ebb563a0f22f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69633::DEBUG::2014-09-25 17:07:53,907::task::993::Storage.TaskManager.Task::(_decref) Task=`6ecbc125-0029-4737-9ac6-ebb563a0f22f`::ref 1 aborting False >Thread-69633::INFO::2014-09-25 17:07:53,911::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69633::DEBUG::2014-09-25 17:07:53,912::task::1191::Storage.TaskManager.Task::(prepare) Task=`6ecbc125-0029-4737-9ac6-ebb563a0f22f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69633::DEBUG::2014-09-25 17:07:53,912::task::595::Storage.TaskManager.Task::(_updateState) Task=`6ecbc125-0029-4737-9ac6-ebb563a0f22f`::moving from state preparing -> state finished >Thread-69633::DEBUG::2014-09-25 17:07:53,912::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69633::DEBUG::2014-09-25 17:07:53,912::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69633::DEBUG::2014-09-25 17:07:53,913::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69633::DEBUG::2014-09-25 17:07:53,913::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69633::DEBUG::2014-09-25 17:07:53,913::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69633::DEBUG::2014-09-25 17:07:53,914::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69633::DEBUG::2014-09-25 17:07:53,914::task::993::Storage.TaskManager.Task::(_decref) Task=`6ecbc125-0029-4737-9ac6-ebb563a0f22f`::ref 0 aborting False >Thread-69633::DEBUG::2014-09-25 17:07:53,914::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75199676416', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69633::DEBUG::2014-09-25 17:07:53,915::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:54,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:54,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194635 s, 52.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:55,624::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:55,624::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:55,624::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:55,624::__init__::232::IOProcess::(_processLogs) (107099) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:55,635::__init__::232::IOProcess::(_processLogs) (107099) Queuing response >Thread-81::DEBUG::2014-09-25 17:07:55,635::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:07:55,667::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00857219 s, 43.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:55,669::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:55,669::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:55,669::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:55,669::__init__::232::IOProcess::(_processLogs) (107100) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:55,676::__init__::232::IOProcess::(_processLogs) (107100) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:55,710::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:55,712::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69634::DEBUG::2014-09-25 17:07:55,714::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:56,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:56,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193255 s, 53.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:58,726::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:58,727::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69635::DEBUG::2014-09-25 17:07:58,730::task::595::Storage.TaskManager.Task::(_updateState) Task=`0de84482-3d83-4a70-a9a3-d439ce556e1a`::moving from state init -> state preparing >Thread-69635::INFO::2014-09-25 17:07:58,730::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69635::INFO::2014-09-25 17:07:58,731::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000456758', 'lastCheck': '8.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00067448', 'lastCheck': '6.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00857219', 'lastCheck': '3.1', 'valid': True}} >Thread-69635::DEBUG::2014-09-25 17:07:58,731::task::1191::Storage.TaskManager.Task::(prepare) Task=`0de84482-3d83-4a70-a9a3-d439ce556e1a`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000456758', 'lastCheck': '8.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00067448', 'lastCheck': '6.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00857219', 'lastCheck': '3.1', 'valid': True}} >Thread-69635::DEBUG::2014-09-25 17:07:58,731::task::595::Storage.TaskManager.Task::(_updateState) Task=`0de84482-3d83-4a70-a9a3-d439ce556e1a`::moving from state preparing -> state finished >Thread-69635::DEBUG::2014-09-25 17:07:58,731::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69635::DEBUG::2014-09-25 17:07:58,732::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69635::DEBUG::2014-09-25 17:07:58,732::task::993::Storage.TaskManager.Task::(_decref) Task=`0de84482-3d83-4a70-a9a3-d439ce556e1a`::ref 0 aborting False >Thread-69635::DEBUG::2014-09-25 17:07:58,734::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:07:58,756::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:07:58,757::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69636::DEBUG::2014-09-25 17:07:58,759::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:07:59,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:07:59,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191311 s, 53.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:07:59,971::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:07:59,972::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:07:59,972::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:07:59,972::__init__::232::IOProcess::(_processLogs) (107101) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:07:59,976::__init__::232::IOProcess::(_processLogs) (107101) Queuing response >Thread-79::DEBUG::2014-09-25 17:07:59,977::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:08:00,014::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00579352 s, 97.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:00,016::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:00,016::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:00,016::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:00,016::__init__::232::IOProcess::(_processLogs) (107102) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:00,019::__init__::232::IOProcess::(_processLogs) (107102) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:00,020::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:00,021::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:00,022::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:00,022::__init__::232::IOProcess::(_processLogs) (107103) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:00,022::__init__::232::IOProcess::(_processLogs) (107103) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:00,023::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:00,023::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:00,024::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:00,024::__init__::232::IOProcess::(_processLogs) (107104) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:00,024::__init__::232::IOProcess::(_processLogs) (107104) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:00,025::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:00,025::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:00,026::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:00,026::__init__::232::IOProcess::(_processLogs) (107105) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:00,027::__init__::232::IOProcess::(_processLogs) (107105) Queuing response >Dummy-148::DEBUG::2014-09-25 17:08:01,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:01,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163992 s, 62.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:01,817::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:01,818::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69637::DEBUG::2014-09-25 17:08:01,820::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:08:01,908::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:01,908::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:01,909::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:01,909::__init__::232::IOProcess::(_processLogs) (107106) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:01,909::__init__::232::IOProcess::(_processLogs) (107106) Queuing response >Thread-80::DEBUG::2014-09-25 17:08:01,911::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:08:01,919::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00050261 s, 752 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:01,921::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:01,921::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:01,921::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:01,921::__init__::232::IOProcess::(_processLogs) (107107) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:01,922::__init__::232::IOProcess::(_processLogs) (107107) Queuing response >Dummy-148::DEBUG::2014-09-25 17:08:03,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:03,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170284 s, 60.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:03,950::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69638::DEBUG::2014-09-25 17:08:03,951::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:08:03,952::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69638::DEBUG::2014-09-25 17:08:03,954::task::595::Storage.TaskManager.Task::(_updateState) Task=`958a3353-36d3-4f73-8050-568bbc9e8747`::moving from state init -> state preparing >Thread-69638::INFO::2014-09-25 17:08:03,954::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69638::INFO::2014-09-25 17:08:03,955::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69638::DEBUG::2014-09-25 17:08:03,955::task::1191::Storage.TaskManager.Task::(prepare) Task=`958a3353-36d3-4f73-8050-568bbc9e8747`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69638::DEBUG::2014-09-25 17:08:03,955::task::595::Storage.TaskManager.Task::(_updateState) Task=`958a3353-36d3-4f73-8050-568bbc9e8747`::moving from state preparing -> state finished >Thread-69638::DEBUG::2014-09-25 17:08:03,956::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69638::DEBUG::2014-09-25 17:08:03,956::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69638::DEBUG::2014-09-25 17:08:03,956::task::993::Storage.TaskManager.Task::(_decref) Task=`958a3353-36d3-4f73-8050-568bbc9e8747`::ref 0 aborting False >Thread-69638::DEBUG::2014-09-25 17:08:03,956::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69638::DEBUG::2014-09-25 17:08:03,957::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:03,968::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:03,969::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69639::DEBUG::2014-09-25 17:08:03,969::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69639::DEBUG::2014-09-25 17:08:03,971::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3a8f82d-95f0-4b32-869e-238712d4c3cb`::moving from state init -> state preparing >Thread-69639::INFO::2014-09-25 17:08:03,971::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69639::DEBUG::2014-09-25 17:08:03,972::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aa72cbc8-d0b5-4327-9a74-d9d96d1f64a4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69639::DEBUG::2014-09-25 17:08:03,972::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69639::DEBUG::2014-09-25 17:08:03,972::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69639::DEBUG::2014-09-25 17:08:03,973::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aa72cbc8-d0b5-4327-9a74-d9d96d1f64a4`::Granted request >Thread-69639::DEBUG::2014-09-25 17:08:03,973::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a3a8f82d-95f0-4b32-869e-238712d4c3cb`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69639::DEBUG::2014-09-25 17:08:03,974::task::993::Storage.TaskManager.Task::(_decref) Task=`a3a8f82d-95f0-4b32-869e-238712d4c3cb`::ref 1 aborting False >Thread-69639::INFO::2014-09-25 17:08:03,978::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69639::DEBUG::2014-09-25 17:08:03,978::task::1191::Storage.TaskManager.Task::(prepare) Task=`a3a8f82d-95f0-4b32-869e-238712d4c3cb`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69639::DEBUG::2014-09-25 17:08:03,978::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3a8f82d-95f0-4b32-869e-238712d4c3cb`::moving from state preparing -> state finished >Thread-69639::DEBUG::2014-09-25 17:08:03,979::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69639::DEBUG::2014-09-25 17:08:03,979::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69639::DEBUG::2014-09-25 17:08:03,979::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69639::DEBUG::2014-09-25 17:08:03,980::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69639::DEBUG::2014-09-25 17:08:03,980::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69639::DEBUG::2014-09-25 17:08:03,980::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69639::DEBUG::2014-09-25 17:08:03,980::task::993::Storage.TaskManager.Task::(_decref) Task=`a3a8f82d-95f0-4b32-869e-238712d4c3cb`::ref 0 aborting False >Thread-69639::DEBUG::2014-09-25 17:08:03,981::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69639::DEBUG::2014-09-25 17:08:03,981::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:04,834::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:04,836::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69640::DEBUG::2014-09-25 17:08:04,837::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:05,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:05,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182781 s, 56.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:05,678::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:05,679::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:05,679::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:05,679::__init__::232::IOProcess::(_processLogs) (107108) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:05,680::__init__::232::IOProcess::(_processLogs) (107108) Queuing response >Thread-81::DEBUG::2014-09-25 17:08:05,681::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:08:05,690::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000474334 s, 782 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:05,691::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:05,691::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:05,691::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:05,692::__init__::232::IOProcess::(_processLogs) (107109) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:05,692::__init__::232::IOProcess::(_processLogs) (107109) Queuing response >Dummy-148::DEBUG::2014-09-25 17:08:07,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:07,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0272709 s, 37.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:07,853::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:07,854::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69641::DEBUG::2014-09-25 17:08:07,856::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:09,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:09,310::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131302 s, 78.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:10,027::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:10,027::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:10,028::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:10,028::__init__::232::IOProcess::(_processLogs) (107110) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:10,043::__init__::232::IOProcess::(_processLogs) (107110) Queuing response >Thread-79::DEBUG::2014-09-25 17:08:10,044::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:08:10,068::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00690458 s, 81.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:10,069::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:10,069::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:10,070::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:10,070::__init__::232::IOProcess::(_processLogs) (107111) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:10,074::__init__::232::IOProcess::(_processLogs) (107111) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:10,075::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:10,076::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:10,077::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:10,077::__init__::232::IOProcess::(_processLogs) (107112) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:10,077::__init__::232::IOProcess::(_processLogs) (107112) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:10,077::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:10,078::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:10,078::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:10,079::__init__::232::IOProcess::(_processLogs) (107113) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:10,079::__init__::232::IOProcess::(_processLogs) (107113) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:10,080::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:10,080::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:10,080::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:10,081::__init__::232::IOProcess::(_processLogs) (107114) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:10,081::__init__::232::IOProcess::(_processLogs) (107114) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:10,877::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:10,879::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69642::DEBUG::2014-09-25 17:08:10,881::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:11,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:11,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174421 s, 58.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:11,924::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:11,925::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:11,925::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:11,925::__init__::232::IOProcess::(_processLogs) (107115) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:11,926::__init__::232::IOProcess::(_processLogs) (107115) Queuing response >Thread-80::DEBUG::2014-09-25 17:08:11,927::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:08:11,936::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000516217 s, 732 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:11,937::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:11,938::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:11,938::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:11,938::__init__::232::IOProcess::(_processLogs) (107116) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:11,939::__init__::232::IOProcess::(_processLogs) (107116) Queuing response >Dummy-148::DEBUG::2014-09-25 17:08:13,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:13,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179606 s, 57.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:13,900::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:13,901::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69643::DEBUG::2014-09-25 17:08:13,904::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ee40b37-2aa5-441e-8391-616512c6e04e`::moving from state init -> state preparing >Thread-69643::INFO::2014-09-25 17:08:13,904::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69643::INFO::2014-09-25 17:08:13,905::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00690458', 'lastCheck': '3.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000516217', 'lastCheck': '2.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000474334', 'lastCheck': '8.2', 'valid': True}} >Thread-69643::DEBUG::2014-09-25 17:08:13,905::task::1191::Storage.TaskManager.Task::(prepare) Task=`8ee40b37-2aa5-441e-8391-616512c6e04e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00690458', 'lastCheck': '3.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000516217', 'lastCheck': '2.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000474334', 'lastCheck': '8.2', 'valid': True}} >Thread-69643::DEBUG::2014-09-25 17:08:13,905::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ee40b37-2aa5-441e-8391-616512c6e04e`::moving from state preparing -> state finished >Thread-69643::DEBUG::2014-09-25 17:08:13,905::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69643::DEBUG::2014-09-25 17:08:13,906::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69643::DEBUG::2014-09-25 17:08:13,906::task::993::Storage.TaskManager.Task::(_decref) Task=`8ee40b37-2aa5-441e-8391-616512c6e04e`::ref 0 aborting False >Thread-69643::DEBUG::2014-09-25 17:08:13,908::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:13,934::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:13,935::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69644::DEBUG::2014-09-25 17:08:13,937::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:14,006::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69645::DEBUG::2014-09-25 17:08:14,007::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:08:14,007::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69645::DEBUG::2014-09-25 17:08:14,009::task::595::Storage.TaskManager.Task::(_updateState) Task=`9caeac70-e18a-496a-a05d-b99939d4ff2c`::moving from state init -> state preparing >Thread-69645::INFO::2014-09-25 17:08:14,009::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69645::INFO::2014-09-25 17:08:14,010::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69645::DEBUG::2014-09-25 17:08:14,010::task::1191::Storage.TaskManager.Task::(prepare) Task=`9caeac70-e18a-496a-a05d-b99939d4ff2c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69645::DEBUG::2014-09-25 17:08:14,010::task::595::Storage.TaskManager.Task::(_updateState) Task=`9caeac70-e18a-496a-a05d-b99939d4ff2c`::moving from state preparing -> state finished >Thread-69645::DEBUG::2014-09-25 17:08:14,010::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69645::DEBUG::2014-09-25 17:08:14,011::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69645::DEBUG::2014-09-25 17:08:14,011::task::993::Storage.TaskManager.Task::(_decref) Task=`9caeac70-e18a-496a-a05d-b99939d4ff2c`::ref 0 aborting False >Thread-69645::DEBUG::2014-09-25 17:08:14,011::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69645::DEBUG::2014-09-25 17:08:14,012::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:14,026::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:14,027::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69646::DEBUG::2014-09-25 17:08:14,027::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69646::DEBUG::2014-09-25 17:08:14,028::task::595::Storage.TaskManager.Task::(_updateState) Task=`b9441288-2395-4395-952d-5db51a0009f7`::moving from state init -> state preparing >Thread-69646::INFO::2014-09-25 17:08:14,029::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69646::DEBUG::2014-09-25 17:08:14,029::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d70527b3-6322-48af-b3f9-365f747b1415`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69646::DEBUG::2014-09-25 17:08:14,030::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69646::DEBUG::2014-09-25 17:08:14,031::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69646::DEBUG::2014-09-25 17:08:14,031::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d70527b3-6322-48af-b3f9-365f747b1415`::Granted request >Thread-69646::DEBUG::2014-09-25 17:08:14,032::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b9441288-2395-4395-952d-5db51a0009f7`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69646::DEBUG::2014-09-25 17:08:14,032::task::993::Storage.TaskManager.Task::(_decref) Task=`b9441288-2395-4395-952d-5db51a0009f7`::ref 1 aborting False >Thread-69646::INFO::2014-09-25 17:08:14,037::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69646::DEBUG::2014-09-25 17:08:14,037::task::1191::Storage.TaskManager.Task::(prepare) Task=`b9441288-2395-4395-952d-5db51a0009f7`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69646::DEBUG::2014-09-25 17:08:14,037::task::595::Storage.TaskManager.Task::(_updateState) Task=`b9441288-2395-4395-952d-5db51a0009f7`::moving from state preparing -> state finished >Thread-69646::DEBUG::2014-09-25 17:08:14,038::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69646::DEBUG::2014-09-25 17:08:14,038::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69646::DEBUG::2014-09-25 17:08:14,038::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69646::DEBUG::2014-09-25 17:08:14,039::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69646::DEBUG::2014-09-25 17:08:14,039::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69646::DEBUG::2014-09-25 17:08:14,039::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69646::DEBUG::2014-09-25 17:08:14,039::task::993::Storage.TaskManager.Task::(_decref) Task=`b9441288-2395-4395-952d-5db51a0009f7`::ref 0 aborting False >Thread-69646::DEBUG::2014-09-25 17:08:14,040::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69646::DEBUG::2014-09-25 17:08:14,040::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:15,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:15,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145776 s, 70.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:15,695::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:15,695::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:15,696::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:08:15,697::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:08:15,697::__init__::232::IOProcess::(_processLogs) (107117) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:15,706::__init__::232::IOProcess::(_processLogs) (107117) Queuing response >Thread-81::DEBUG::2014-09-25 17:08:15,707::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000541768 s, 685 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:15,708::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:15,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:15,708::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:15,709::__init__::232::IOProcess::(_processLogs) (107118) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:15,709::__init__::232::IOProcess::(_processLogs) (107118) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:16,999::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:17,001::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69647::DEBUG::2014-09-25 17:08:17,005::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:17,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:17,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160745 s, 63.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:08:19,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:19,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212834 s, 48.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:20,024::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:20,025::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69648::DEBUG::2014-09-25 17:08:20,027::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:08:20,082::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:20,082::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:20,082::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:20,082::__init__::232::IOProcess::(_processLogs) (107119) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:20,086::__init__::232::IOProcess::(_processLogs) (107119) Queuing response >Thread-79::DEBUG::2014-09-25 17:08:20,087::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:08:20,106::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00753991 s, 74.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:20,107::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:20,108::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:20,108::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:20,108::__init__::232::IOProcess::(_processLogs) (107120) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:20,116::__init__::232::IOProcess::(_processLogs) (107120) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:20,117::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:20,117::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:20,117::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:20,118::__init__::232::IOProcess::(_processLogs) (107121) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:20,119::__init__::232::IOProcess::(_processLogs) (107121) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:20,119::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:20,120::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:20,120::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:20,120::__init__::232::IOProcess::(_processLogs) (107122) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:20,121::__init__::232::IOProcess::(_processLogs) (107122) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:20,121::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:20,121::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:20,122::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:20,123::__init__::232::IOProcess::(_processLogs) (107123) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:20,123::__init__::232::IOProcess::(_processLogs) (107123) Queuing response >Dummy-148::DEBUG::2014-09-25 17:08:21,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:21,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020457 s, 50.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:21,941::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:21,941::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:21,942::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:21,942::__init__::232::IOProcess::(_processLogs) (107124) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:21,943::__init__::232::IOProcess::(_processLogs) (107124) Queuing response >Thread-80::DEBUG::2014-09-25 17:08:21,943::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:08:21,953::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00051484 s, 734 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:21,954::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:21,954::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:21,954::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:21,955::__init__::232::IOProcess::(_processLogs) (107125) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:21,955::__init__::232::IOProcess::(_processLogs) (107125) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:23,050::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:23,052::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69649::DEBUG::2014-09-25 17:08:23,055::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:23,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:23,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143226 s, 71.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:24,069::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69650::DEBUG::2014-09-25 17:08:24,070::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:08:24,070::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69650::DEBUG::2014-09-25 17:08:24,072::task::595::Storage.TaskManager.Task::(_updateState) Task=`25ade5b2-eead-4a77-b87e-04a8167e78d3`::moving from state init -> state preparing >Thread-69650::INFO::2014-09-25 17:08:24,073::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69650::INFO::2014-09-25 17:08:24,073::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69650::DEBUG::2014-09-25 17:08:24,074::task::1191::Storage.TaskManager.Task::(prepare) Task=`25ade5b2-eead-4a77-b87e-04a8167e78d3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69650::DEBUG::2014-09-25 17:08:24,074::task::595::Storage.TaskManager.Task::(_updateState) Task=`25ade5b2-eead-4a77-b87e-04a8167e78d3`::moving from state preparing -> state finished >Thread-69650::DEBUG::2014-09-25 17:08:24,074::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69650::DEBUG::2014-09-25 17:08:24,074::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69650::DEBUG::2014-09-25 17:08:24,074::task::993::Storage.TaskManager.Task::(_decref) Task=`25ade5b2-eead-4a77-b87e-04a8167e78d3`::ref 0 aborting False >Thread-69650::DEBUG::2014-09-25 17:08:24,075::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69650::DEBUG::2014-09-25 17:08:24,075::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:24,089::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:24,091::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69651::DEBUG::2014-09-25 17:08:24,091::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69651::DEBUG::2014-09-25 17:08:24,092::task::595::Storage.TaskManager.Task::(_updateState) Task=`a1518689-a2e8-40a8-a6f5-773108dbe41a`::moving from state init -> state preparing >Thread-69651::INFO::2014-09-25 17:08:24,093::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69651::DEBUG::2014-09-25 17:08:24,093::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`bddfcc56-d65e-4f61-8c5f-dceac85f6547`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69651::DEBUG::2014-09-25 17:08:24,094::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69651::DEBUG::2014-09-25 17:08:24,094::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69651::DEBUG::2014-09-25 17:08:24,094::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`bddfcc56-d65e-4f61-8c5f-dceac85f6547`::Granted request >Thread-69651::DEBUG::2014-09-25 17:08:24,095::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a1518689-a2e8-40a8-a6f5-773108dbe41a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69651::DEBUG::2014-09-25 17:08:24,095::task::993::Storage.TaskManager.Task::(_decref) Task=`a1518689-a2e8-40a8-a6f5-773108dbe41a`::ref 1 aborting False >Thread-69651::INFO::2014-09-25 17:08:24,099::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69651::DEBUG::2014-09-25 17:08:24,100::task::1191::Storage.TaskManager.Task::(prepare) Task=`a1518689-a2e8-40a8-a6f5-773108dbe41a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69651::DEBUG::2014-09-25 17:08:24,100::task::595::Storage.TaskManager.Task::(_updateState) Task=`a1518689-a2e8-40a8-a6f5-773108dbe41a`::moving from state preparing -> state finished >Thread-69651::DEBUG::2014-09-25 17:08:24,100::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69651::DEBUG::2014-09-25 17:08:24,100::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69651::DEBUG::2014-09-25 17:08:24,101::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69651::DEBUG::2014-09-25 17:08:24,101::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69651::DEBUG::2014-09-25 17:08:24,101::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69651::DEBUG::2014-09-25 17:08:24,102::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69651::DEBUG::2014-09-25 17:08:24,102::task::993::Storage.TaskManager.Task::(_decref) Task=`a1518689-a2e8-40a8-a6f5-773108dbe41a`::ref 0 aborting False >Thread-69651::DEBUG::2014-09-25 17:08:24,102::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69651::DEBUG::2014-09-25 17:08:24,103::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:25,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:08:25,712::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:25,714::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:25,715::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:25,716::__init__::232::IOProcess::(_processLogs) (107126) Got request for method 'statvfs' >Dummy-148::DEBUG::2014-09-25 17:08:25,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0258884 s, 39.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:25,721::__init__::232::IOProcess::(_processLogs) (107126) Queuing response >Thread-81::DEBUG::2014-09-25 17:08:25,724::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:08:25,763::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00896013 s, 41.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:25,764::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:25,764::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:25,765::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:25,765::__init__::232::IOProcess::(_processLogs) (107127) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:25,767::__init__::232::IOProcess::(_processLogs) (107127) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:26,074::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:26,075::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69652::DEBUG::2014-09-25 17:08:26,077::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:27,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:27,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132477 s, 77.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:29,093::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:29,095::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69653::DEBUG::2014-09-25 17:08:29,097::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e39eb64-dca1-4cb5-b231-68e824b85b5a`::moving from state init -> state preparing >Thread-69653::INFO::2014-09-25 17:08:29,098::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69653::INFO::2014-09-25 17:08:29,098::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00753991', 'lastCheck': '9.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00051484', 'lastCheck': '7.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00896013', 'lastCheck': '3.3', 'valid': True}} >Thread-69653::DEBUG::2014-09-25 17:08:29,098::task::1191::Storage.TaskManager.Task::(prepare) Task=`3e39eb64-dca1-4cb5-b231-68e824b85b5a`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00753991', 'lastCheck': '9.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00051484', 'lastCheck': '7.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00896013', 'lastCheck': '3.3', 'valid': True}} >Thread-69653::DEBUG::2014-09-25 17:08:29,099::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e39eb64-dca1-4cb5-b231-68e824b85b5a`::moving from state preparing -> state finished >Thread-69653::DEBUG::2014-09-25 17:08:29,099::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69653::DEBUG::2014-09-25 17:08:29,099::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69653::DEBUG::2014-09-25 17:08:29,099::task::993::Storage.TaskManager.Task::(_decref) Task=`3e39eb64-dca1-4cb5-b231-68e824b85b5a`::ref 0 aborting False >Thread-69653::DEBUG::2014-09-25 17:08:29,102::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:29,127::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:29,131::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69654::DEBUG::2014-09-25 17:08:29,135::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:29,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:29,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189406 s, 54.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:30,124::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:30,124::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:30,124::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:30,125::__init__::232::IOProcess::(_processLogs) (107128) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:30,130::__init__::232::IOProcess::(_processLogs) (107128) Queuing response >Thread-79::DEBUG::2014-09-25 17:08:30,131::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:08:30,145::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00388706 s, 145 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:30,146::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:30,147::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:30,147::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:30,147::__init__::232::IOProcess::(_processLogs) (107129) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:30,151::__init__::232::IOProcess::(_processLogs) (107129) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:30,153::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:30,153::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:30,153::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:30,154::__init__::232::IOProcess::(_processLogs) (107130) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:30,154::__init__::232::IOProcess::(_processLogs) (107130) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:30,155::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:30,155::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:30,155::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:30,155::__init__::232::IOProcess::(_processLogs) (107131) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:30,156::__init__::232::IOProcess::(_processLogs) (107131) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:30,157::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:30,157::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:30,157::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:30,158::__init__::232::IOProcess::(_processLogs) (107132) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:30,159::__init__::232::IOProcess::(_processLogs) (107132) Queuing response >Dummy-148::DEBUG::2014-09-25 17:08:31,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:31,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170862 s, 59.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:31,958::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:31,958::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:31,958::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:31,958::__init__::232::IOProcess::(_processLogs) (107133) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:31,959::__init__::232::IOProcess::(_processLogs) (107133) Queuing response >Thread-80::DEBUG::2014-09-25 17:08:31,960::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:08:31,969::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000549758 s, 688 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:31,970::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:31,970::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:31,971::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:31,971::__init__::232::IOProcess::(_processLogs) (107134) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:31,972::__init__::232::IOProcess::(_processLogs) (107134) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:32,204::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:32,205::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69655::DEBUG::2014-09-25 17:08:32,207::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:33,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:33,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133331 s, 76.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:34,136::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:34,138::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69656::DEBUG::2014-09-25 17:08:34,138::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69656::DEBUG::2014-09-25 17:08:34,140::task::595::Storage.TaskManager.Task::(_updateState) Task=`48458222-990d-43db-b56e-46b9b96a0a2a`::moving from state init -> state preparing >Thread-69656::INFO::2014-09-25 17:08:34,140::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69656::INFO::2014-09-25 17:08:34,141::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69656::DEBUG::2014-09-25 17:08:34,141::task::1191::Storage.TaskManager.Task::(prepare) Task=`48458222-990d-43db-b56e-46b9b96a0a2a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69656::DEBUG::2014-09-25 17:08:34,141::task::595::Storage.TaskManager.Task::(_updateState) Task=`48458222-990d-43db-b56e-46b9b96a0a2a`::moving from state preparing -> state finished >Thread-69656::DEBUG::2014-09-25 17:08:34,141::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69656::DEBUG::2014-09-25 17:08:34,142::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69656::DEBUG::2014-09-25 17:08:34,142::task::993::Storage.TaskManager.Task::(_decref) Task=`48458222-990d-43db-b56e-46b9b96a0a2a`::ref 0 aborting False >Thread-69656::DEBUG::2014-09-25 17:08:34,142::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69656::DEBUG::2014-09-25 17:08:34,143::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:34,158::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:34,159::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69657::DEBUG::2014-09-25 17:08:34,159::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69657::DEBUG::2014-09-25 17:08:34,161::task::595::Storage.TaskManager.Task::(_updateState) Task=`df417a62-214d-4a17-be06-2ebe833a1981`::moving from state init -> state preparing >Thread-69657::INFO::2014-09-25 17:08:34,161::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69657::DEBUG::2014-09-25 17:08:34,162::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e4583f7b-d0c2-4afc-8078-1e4e8c32461f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69657::DEBUG::2014-09-25 17:08:34,162::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69657::DEBUG::2014-09-25 17:08:34,163::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69657::DEBUG::2014-09-25 17:08:34,163::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e4583f7b-d0c2-4afc-8078-1e4e8c32461f`::Granted request >Thread-69657::DEBUG::2014-09-25 17:08:34,164::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`df417a62-214d-4a17-be06-2ebe833a1981`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69657::DEBUG::2014-09-25 17:08:34,164::task::993::Storage.TaskManager.Task::(_decref) Task=`df417a62-214d-4a17-be06-2ebe833a1981`::ref 1 aborting False >Thread-69657::INFO::2014-09-25 17:08:34,168::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69657::DEBUG::2014-09-25 17:08:34,168::task::1191::Storage.TaskManager.Task::(prepare) Task=`df417a62-214d-4a17-be06-2ebe833a1981`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69657::DEBUG::2014-09-25 17:08:34,169::task::595::Storage.TaskManager.Task::(_updateState) Task=`df417a62-214d-4a17-be06-2ebe833a1981`::moving from state preparing -> state finished >Thread-69657::DEBUG::2014-09-25 17:08:34,169::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69657::DEBUG::2014-09-25 17:08:34,169::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69657::DEBUG::2014-09-25 17:08:34,169::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69657::DEBUG::2014-09-25 17:08:34,170::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69657::DEBUG::2014-09-25 17:08:34,170::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69657::DEBUG::2014-09-25 17:08:34,170::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69657::DEBUG::2014-09-25 17:08:34,171::task::993::Storage.TaskManager.Task::(_decref) Task=`df417a62-214d-4a17-be06-2ebe833a1981`::ref 0 aborting False >Thread-69657::DEBUG::2014-09-25 17:08:34,171::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69657::DEBUG::2014-09-25 17:08:34,172::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:35,224::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:35,225::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69658::DEBUG::2014-09-25 17:08:35,227::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:08:35,769::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:35,770::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:35,770::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:35,770::__init__::232::IOProcess::(_processLogs) (107135) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:35,774::__init__::232::IOProcess::(_processLogs) (107135) Queuing response >Thread-81::DEBUG::2014-09-25 17:08:35,775::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:08:35,796::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00837625 s, 44.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:35,797::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:35,797::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:35,798::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:35,798::__init__::232::IOProcess::(_processLogs) (107136) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:35,802::__init__::232::IOProcess::(_processLogs) (107136) Queuing response >Dummy-148::DEBUG::2014-09-25 17:08:35,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:35,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178629 s, 57.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:08:37,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:38,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0251058 s, 40.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:38,242::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:38,243::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69659::DEBUG::2014-09-25 17:08:38,245::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:40,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:40,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151099 s, 67.8 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 17:08:40,158::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:08:40,163::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,164::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,164::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,165::__init__::232::IOProcess::(_processLogs) (107137) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:08:40,165::__init__::232::IOProcess::(_processLogs) (107137) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:40,166::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,166::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,167::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,167::__init__::232::IOProcess::(_processLogs) (107138) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:08:40,167::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,168::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,169::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,169::__init__::232::IOProcess::(_processLogs) (107139) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:08:40,169::__init__::232::IOProcess::(_processLogs) (107138) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:40,169::__init__::232::IOProcess::(_processLogs) (107139) Queuing response >Thread-79::DEBUG::2014-09-25 17:08:40,173::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:08:40,174::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,174::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,174::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,175::__init__::232::IOProcess::(_processLogs) (107140) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:08:40,191::__init__::232::IOProcess::(_processLogs) (107140) Queuing response >Thread-79::DEBUG::2014-09-25 17:08:40,191::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:08:40,193::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,193::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,193::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,193::__init__::232::IOProcess::(_processLogs) (107141) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:08:40,194::__init__::232::IOProcess::(_processLogs) (107141) Queuing response >Thread-79::DEBUG::2014-09-25 17:08:40,194::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:08:40,196::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,197::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,197::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,197::__init__::232::IOProcess::(_processLogs) (107142) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:40,198::__init__::232::IOProcess::(_processLogs) (107142) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:40,198::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,198::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,198::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,199::__init__::232::IOProcess::(_processLogs) (107143) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:08:40,199::__init__::232::IOProcess::(_processLogs) (107143) Queuing response >Thread-79::DEBUG::2014-09-25 17:08:40,199::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 17:08:40,200::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:08:40,201::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:08:40,201::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,201::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,202::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,202::__init__::232::IOProcess::(_processLogs) (107144) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:40,203::__init__::232::IOProcess::(_processLogs) (107144) Queuing response >Thread-79::DEBUG::2014-09-25 17:08:40,203::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:08:40,212::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000365622 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:40,213::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,213::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,214::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,214::__init__::232::IOProcess::(_processLogs) (107145) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:40,215::__init__::232::IOProcess::(_processLogs) (107145) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:40,216::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,216::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,216::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,217::__init__::232::IOProcess::(_processLogs) (107146) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:40,217::__init__::232::IOProcess::(_processLogs) (107146) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:40,218::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,218::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,218::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,219::__init__::232::IOProcess::(_processLogs) (107147) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:40,219::__init__::232::IOProcess::(_processLogs) (107147) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:40,220::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:40,220::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:40,221::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:40,222::__init__::232::IOProcess::(_processLogs) (107148) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:40,222::__init__::232::IOProcess::(_processLogs) (107148) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:41,263::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:41,264::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69663::DEBUG::2014-09-25 17:08:41,266::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:08:41,974::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:41,974::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:41,975::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:41,975::__init__::232::IOProcess::(_processLogs) (107149) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:41,976::__init__::232::IOProcess::(_processLogs) (107149) Queuing response >Thread-80::DEBUG::2014-09-25 17:08:41,977::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:08:41,986::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000535926 s, 705 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:41,987::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:41,988::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:41,988::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:41,988::__init__::232::IOProcess::(_processLogs) (107150) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:41,988::__init__::232::IOProcess::(_processLogs) (107150) Queuing response >Dummy-148::DEBUG::2014-09-25 17:08:42,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:42,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130891 s, 78.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:08:44,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:44,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161853 s, 63.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:44,214::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:44,216::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69664::DEBUG::2014-09-25 17:08:44,216::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69664::DEBUG::2014-09-25 17:08:44,218::task::595::Storage.TaskManager.Task::(_updateState) Task=`2da081fa-6478-487c-a69c-81ce9f9e82dc`::moving from state init -> state preparing >Thread-69664::INFO::2014-09-25 17:08:44,219::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69664::INFO::2014-09-25 17:08:44,219::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69664::DEBUG::2014-09-25 17:08:44,219::task::1191::Storage.TaskManager.Task::(prepare) Task=`2da081fa-6478-487c-a69c-81ce9f9e82dc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69664::DEBUG::2014-09-25 17:08:44,220::task::595::Storage.TaskManager.Task::(_updateState) Task=`2da081fa-6478-487c-a69c-81ce9f9e82dc`::moving from state preparing -> state finished >Thread-69664::DEBUG::2014-09-25 17:08:44,220::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69664::DEBUG::2014-09-25 17:08:44,220::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69664::DEBUG::2014-09-25 17:08:44,220::task::993::Storage.TaskManager.Task::(_decref) Task=`2da081fa-6478-487c-a69c-81ce9f9e82dc`::ref 0 aborting False >Thread-69664::DEBUG::2014-09-25 17:08:44,221::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69664::DEBUG::2014-09-25 17:08:44,221::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:44,242::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:44,243::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69665::DEBUG::2014-09-25 17:08:44,243::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69665::DEBUG::2014-09-25 17:08:44,245::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb9470bb-8aea-41b7-a3c9-3bb10574e424`::moving from state init -> state preparing >Thread-69665::INFO::2014-09-25 17:08:44,245::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69665::DEBUG::2014-09-25 17:08:44,246::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a3185244-863e-4546-bb48-4a15e0827cc7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69665::DEBUG::2014-09-25 17:08:44,246::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69665::DEBUG::2014-09-25 17:08:44,247::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69665::DEBUG::2014-09-25 17:08:44,247::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a3185244-863e-4546-bb48-4a15e0827cc7`::Granted request >Thread-69665::DEBUG::2014-09-25 17:08:44,248::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`fb9470bb-8aea-41b7-a3c9-3bb10574e424`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69665::DEBUG::2014-09-25 17:08:44,248::task::993::Storage.TaskManager.Task::(_decref) Task=`fb9470bb-8aea-41b7-a3c9-3bb10574e424`::ref 1 aborting False >Thread-69665::INFO::2014-09-25 17:08:44,252::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69665::DEBUG::2014-09-25 17:08:44,253::task::1191::Storage.TaskManager.Task::(prepare) Task=`fb9470bb-8aea-41b7-a3c9-3bb10574e424`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69665::DEBUG::2014-09-25 17:08:44,253::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb9470bb-8aea-41b7-a3c9-3bb10574e424`::moving from state preparing -> state finished >Thread-69665::DEBUG::2014-09-25 17:08:44,253::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69665::DEBUG::2014-09-25 17:08:44,253::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69665::DEBUG::2014-09-25 17:08:44,254::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69665::DEBUG::2014-09-25 17:08:44,254::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69665::DEBUG::2014-09-25 17:08:44,254::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69665::DEBUG::2014-09-25 17:08:44,254::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69665::DEBUG::2014-09-25 17:08:44,255::task::993::Storage.TaskManager.Task::(_decref) Task=`fb9470bb-8aea-41b7-a3c9-3bb10574e424`::ref 0 aborting False >Thread-69665::DEBUG::2014-09-25 17:08:44,255::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69665::DEBUG::2014-09-25 17:08:44,256::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:44,280::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:44,281::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69666::DEBUG::2014-09-25 17:08:44,283::task::595::Storage.TaskManager.Task::(_updateState) Task=`70b95ed4-8b00-4d34-bf06-954ad00075d7`::moving from state init -> state preparing >Thread-69666::INFO::2014-09-25 17:08:44,283::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69666::INFO::2014-09-25 17:08:44,284::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000365622', 'lastCheck': '4.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000535926', 'lastCheck': '2.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00837625', 'lastCheck': '8.5', 'valid': True}} >Thread-69666::DEBUG::2014-09-25 17:08:44,284::task::1191::Storage.TaskManager.Task::(prepare) Task=`70b95ed4-8b00-4d34-bf06-954ad00075d7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000365622', 'lastCheck': '4.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000535926', 'lastCheck': '2.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00837625', 'lastCheck': '8.5', 'valid': True}} >Thread-69666::DEBUG::2014-09-25 17:08:44,284::task::595::Storage.TaskManager.Task::(_updateState) Task=`70b95ed4-8b00-4d34-bf06-954ad00075d7`::moving from state preparing -> state finished >Thread-69666::DEBUG::2014-09-25 17:08:44,285::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69666::DEBUG::2014-09-25 17:08:44,285::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69666::DEBUG::2014-09-25 17:08:44,285::task::993::Storage.TaskManager.Task::(_decref) Task=`70b95ed4-8b00-4d34-bf06-954ad00075d7`::ref 0 aborting False >Thread-69666::DEBUG::2014-09-25 17:08:44,287::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:44,311::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:44,312::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69667::DEBUG::2014-09-25 17:08:44,314::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:08:45,805::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:45,805::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:45,805::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:45,806::__init__::232::IOProcess::(_processLogs) (107151) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:45,807::__init__::232::IOProcess::(_processLogs) (107151) Queuing response >Thread-81::DEBUG::2014-09-25 17:08:45,808::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:08:45,826::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00782669 s, 47.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:45,827::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:45,827::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:45,827::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:45,828::__init__::232::IOProcess::(_processLogs) (107152) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:45,829::__init__::232::IOProcess::(_processLogs) (107152) Queuing response >Dummy-148::DEBUG::2014-09-25 17:08:46,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:46,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0226792 s, 45.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:47,378::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:47,379::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69668::DEBUG::2014-09-25 17:08:47,381::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:48,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:48,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136725 s, 74.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:50,222::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:50,223::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:50,223::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:50,223::__init__::232::IOProcess::(_processLogs) (107153) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:50,225::__init__::232::IOProcess::(_processLogs) (107153) Queuing response >Thread-79::DEBUG::2014-09-25 17:08:50,226::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:08:50,247::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00642158 s, 87.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:50,248::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:50,248::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:50,248::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:50,249::__init__::232::IOProcess::(_processLogs) (107154) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:50,250::__init__::232::IOProcess::(_processLogs) (107154) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:50,251::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:50,251::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:50,251::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:50,252::__init__::232::IOProcess::(_processLogs) (107155) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:50,252::__init__::232::IOProcess::(_processLogs) (107155) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:50,253::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:50,253::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:50,253::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:50,254::__init__::232::IOProcess::(_processLogs) (107156) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:50,254::__init__::232::IOProcess::(_processLogs) (107156) Queuing response >Thread-29::DEBUG::2014-09-25 17:08:50,255::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:50,255::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:50,256::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:50,257::__init__::232::IOProcess::(_processLogs) (107157) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:08:50,257::__init__::232::IOProcess::(_processLogs) (107157) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:50,396::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:50,397::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69669::DEBUG::2014-09-25 17:08:50,400::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:50,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:50,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158107 s, 64.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:51,991::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:51,992::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:51,992::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:51,992::__init__::232::IOProcess::(_processLogs) (107158) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:51,992::__init__::232::IOProcess::(_processLogs) (107158) Queuing response >Thread-80::DEBUG::2014-09-25 17:08:51,993::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:08:52,002::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000582224 s, 649 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:52,003::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:52,003::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:52,004::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:52,005::__init__::232::IOProcess::(_processLogs) (107159) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:52,005::__init__::232::IOProcess::(_processLogs) (107159) Queuing response >Dummy-148::DEBUG::2014-09-25 17:08:52,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:52,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133613 s, 76.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:53,416::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:53,417::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69670::DEBUG::2014-09-25 17:08:53,419::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:54,291::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69671::DEBUG::2014-09-25 17:08:54,292::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:08:54,292::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69671::DEBUG::2014-09-25 17:08:54,294::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e022ec7-8fc6-49b5-b3a7-023d215909c3`::moving from state init -> state preparing >Thread-69671::INFO::2014-09-25 17:08:54,295::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69671::INFO::2014-09-25 17:08:54,295::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69671::DEBUG::2014-09-25 17:08:54,295::task::1191::Storage.TaskManager.Task::(prepare) Task=`3e022ec7-8fc6-49b5-b3a7-023d215909c3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69671::DEBUG::2014-09-25 17:08:54,296::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e022ec7-8fc6-49b5-b3a7-023d215909c3`::moving from state preparing -> state finished >Thread-69671::DEBUG::2014-09-25 17:08:54,296::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69671::DEBUG::2014-09-25 17:08:54,296::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69671::DEBUG::2014-09-25 17:08:54,296::task::993::Storage.TaskManager.Task::(_decref) Task=`3e022ec7-8fc6-49b5-b3a7-023d215909c3`::ref 0 aborting False >Thread-69671::DEBUG::2014-09-25 17:08:54,297::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69671::DEBUG::2014-09-25 17:08:54,297::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:54,311::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:54,312::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69672::DEBUG::2014-09-25 17:08:54,312::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69672::DEBUG::2014-09-25 17:08:54,314::task::595::Storage.TaskManager.Task::(_updateState) Task=`adf11069-ce1e-434b-ac64-1ff5d700cfd8`::moving from state init -> state preparing >Thread-69672::INFO::2014-09-25 17:08:54,314::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69672::DEBUG::2014-09-25 17:08:54,315::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`33e4b288-2c00-4b0e-91db-90b34e2e6596`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69672::DEBUG::2014-09-25 17:08:54,315::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69672::DEBUG::2014-09-25 17:08:54,315::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69672::DEBUG::2014-09-25 17:08:54,316::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`33e4b288-2c00-4b0e-91db-90b34e2e6596`::Granted request >Thread-69672::DEBUG::2014-09-25 17:08:54,317::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`adf11069-ce1e-434b-ac64-1ff5d700cfd8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69672::DEBUG::2014-09-25 17:08:54,318::task::993::Storage.TaskManager.Task::(_decref) Task=`adf11069-ce1e-434b-ac64-1ff5d700cfd8`::ref 1 aborting False >Thread-69672::INFO::2014-09-25 17:08:54,322::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69672::DEBUG::2014-09-25 17:08:54,323::task::1191::Storage.TaskManager.Task::(prepare) Task=`adf11069-ce1e-434b-ac64-1ff5d700cfd8`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69672::DEBUG::2014-09-25 17:08:54,323::task::595::Storage.TaskManager.Task::(_updateState) Task=`adf11069-ce1e-434b-ac64-1ff5d700cfd8`::moving from state preparing -> state finished >Thread-69672::DEBUG::2014-09-25 17:08:54,323::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69672::DEBUG::2014-09-25 17:08:54,323::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69672::DEBUG::2014-09-25 17:08:54,324::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69672::DEBUG::2014-09-25 17:08:54,324::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69672::DEBUG::2014-09-25 17:08:54,324::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69672::DEBUG::2014-09-25 17:08:54,325::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69672::DEBUG::2014-09-25 17:08:54,325::task::993::Storage.TaskManager.Task::(_decref) Task=`adf11069-ce1e-434b-ac64-1ff5d700cfd8`::ref 0 aborting False >Thread-69672::DEBUG::2014-09-25 17:08:54,325::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69672::DEBUG::2014-09-25 17:08:54,326::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:54,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:54,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174042 s, 58.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:55,831::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:55,831::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:55,831::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:55,832::__init__::232::IOProcess::(_processLogs) (107160) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:55,845::__init__::232::IOProcess::(_processLogs) (107160) Queuing response >Thread-81::DEBUG::2014-09-25 17:08:55,846::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:08:55,864::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0024424 s, 152 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:08:55,866::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:08:55,866::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:08:55,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:08:55,867::__init__::232::IOProcess::(_processLogs) (107161) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:08:55,872::__init__::232::IOProcess::(_processLogs) (107161) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:56,438::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:56,439::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69673::DEBUG::2014-09-25 17:08:56,441::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:08:56,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:56,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.023063 s, 44.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:08:58,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:08:58,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166395 s, 61.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:59,458::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:59,459::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69674::DEBUG::2014-09-25 17:08:59,461::task::595::Storage.TaskManager.Task::(_updateState) Task=`45233e29-67b8-4d18-b627-b1e6204b58f6`::moving from state init -> state preparing >Thread-69674::INFO::2014-09-25 17:08:59,462::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69674::INFO::2014-09-25 17:08:59,462::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00642158', 'lastCheck': '9.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000582224', 'lastCheck': '7.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0024424', 'lastCheck': '3.6', 'valid': True}} >Thread-69674::DEBUG::2014-09-25 17:08:59,462::task::1191::Storage.TaskManager.Task::(prepare) Task=`45233e29-67b8-4d18-b627-b1e6204b58f6`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00642158', 'lastCheck': '9.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000582224', 'lastCheck': '7.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0024424', 'lastCheck': '3.6', 'valid': True}} >Thread-69674::DEBUG::2014-09-25 17:08:59,463::task::595::Storage.TaskManager.Task::(_updateState) Task=`45233e29-67b8-4d18-b627-b1e6204b58f6`::moving from state preparing -> state finished >Thread-69674::DEBUG::2014-09-25 17:08:59,463::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69674::DEBUG::2014-09-25 17:08:59,463::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69674::DEBUG::2014-09-25 17:08:59,463::task::993::Storage.TaskManager.Task::(_decref) Task=`45233e29-67b8-4d18-b627-b1e6204b58f6`::ref 0 aborting False >Thread-69674::DEBUG::2014-09-25 17:08:59,466::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:08:59,488::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:08:59,489::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69675::DEBUG::2014-09-25 17:08:59,491::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:09:00,257::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:00,257::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:00,258::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:00,258::__init__::232::IOProcess::(_processLogs) (107162) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:00,258::__init__::232::IOProcess::(_processLogs) (107162) Queuing response >Thread-79::DEBUG::2014-09-25 17:09:00,260::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:09:00,269::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000477718 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:00,270::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:00,270::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:00,271::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:00,271::__init__::232::IOProcess::(_processLogs) (107163) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:00,271::__init__::232::IOProcess::(_processLogs) (107163) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:00,273::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:00,273::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:00,274::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:00,274::__init__::232::IOProcess::(_processLogs) (107164) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:00,274::__init__::232::IOProcess::(_processLogs) (107164) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:00,275::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:00,275::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:00,276::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:00,276::__init__::232::IOProcess::(_processLogs) (107165) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:00,276::__init__::232::IOProcess::(_processLogs) (107165) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:00,277::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:00,277::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:00,278::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:00,278::__init__::232::IOProcess::(_processLogs) (107166) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:00,279::__init__::232::IOProcess::(_processLogs) (107166) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:00,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:00,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021447 s, 47.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:02,006::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:02,007::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:02,007::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:02,007::__init__::232::IOProcess::(_processLogs) (107167) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:02,007::__init__::232::IOProcess::(_processLogs) (107167) Queuing response >Thread-80::DEBUG::2014-09-25 17:09:02,009::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:09:02,018::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000680398 s, 556 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:02,019::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:02,019::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:02,020::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:02,020::__init__::232::IOProcess::(_processLogs) (107168) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:02,021::__init__::232::IOProcess::(_processLogs) (107168) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:02,554::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:02,555::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69676::DEBUG::2014-09-25 17:09:02,557::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:02,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:02,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138087 s, 74.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:04,357::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69677::DEBUG::2014-09-25 17:09:04,358::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:09:04,358::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69677::DEBUG::2014-09-25 17:09:04,361::task::595::Storage.TaskManager.Task::(_updateState) Task=`38e59370-d42f-4ed8-9ca5-fe70ba631389`::moving from state init -> state preparing >Thread-69677::INFO::2014-09-25 17:09:04,361::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69677::INFO::2014-09-25 17:09:04,362::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69677::DEBUG::2014-09-25 17:09:04,362::task::1191::Storage.TaskManager.Task::(prepare) Task=`38e59370-d42f-4ed8-9ca5-fe70ba631389`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69677::DEBUG::2014-09-25 17:09:04,362::task::595::Storage.TaskManager.Task::(_updateState) Task=`38e59370-d42f-4ed8-9ca5-fe70ba631389`::moving from state preparing -> state finished >Thread-69677::DEBUG::2014-09-25 17:09:04,363::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69677::DEBUG::2014-09-25 17:09:04,363::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69677::DEBUG::2014-09-25 17:09:04,363::task::993::Storage.TaskManager.Task::(_decref) Task=`38e59370-d42f-4ed8-9ca5-fe70ba631389`::ref 0 aborting False >Thread-69677::DEBUG::2014-09-25 17:09:04,363::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69677::DEBUG::2014-09-25 17:09:04,364::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:04,379::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:04,380::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69678::DEBUG::2014-09-25 17:09:04,380::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69678::DEBUG::2014-09-25 17:09:04,382::task::595::Storage.TaskManager.Task::(_updateState) Task=`be21d1eb-57ac-441b-8085-2a3f39d80837`::moving from state init -> state preparing >Thread-69678::INFO::2014-09-25 17:09:04,382::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69678::DEBUG::2014-09-25 17:09:04,383::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a40ff81c-4e4d-41b1-84d6-6e2e3c56fc37`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69678::DEBUG::2014-09-25 17:09:04,383::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69678::DEBUG::2014-09-25 17:09:04,383::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69678::DEBUG::2014-09-25 17:09:04,383::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a40ff81c-4e4d-41b1-84d6-6e2e3c56fc37`::Granted request >Thread-69678::DEBUG::2014-09-25 17:09:04,384::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`be21d1eb-57ac-441b-8085-2a3f39d80837`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69678::DEBUG::2014-09-25 17:09:04,384::task::993::Storage.TaskManager.Task::(_decref) Task=`be21d1eb-57ac-441b-8085-2a3f39d80837`::ref 1 aborting False >Thread-69678::INFO::2014-09-25 17:09:04,389::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69678::DEBUG::2014-09-25 17:09:04,389::task::1191::Storage.TaskManager.Task::(prepare) Task=`be21d1eb-57ac-441b-8085-2a3f39d80837`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69678::DEBUG::2014-09-25 17:09:04,389::task::595::Storage.TaskManager.Task::(_updateState) Task=`be21d1eb-57ac-441b-8085-2a3f39d80837`::moving from state preparing -> state finished >Thread-69678::DEBUG::2014-09-25 17:09:04,389::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69678::DEBUG::2014-09-25 17:09:04,390::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69678::DEBUG::2014-09-25 17:09:04,390::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69678::DEBUG::2014-09-25 17:09:04,390::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69678::DEBUG::2014-09-25 17:09:04,391::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69678::DEBUG::2014-09-25 17:09:04,391::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69678::DEBUG::2014-09-25 17:09:04,391::task::993::Storage.TaskManager.Task::(_decref) Task=`be21d1eb-57ac-441b-8085-2a3f39d80837`::ref 0 aborting False >Thread-69678::DEBUG::2014-09-25 17:09:04,392::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69678::DEBUG::2014-09-25 17:09:04,392::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:04,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:05,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.220499 s, 4.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:05,575::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:05,576::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69679::DEBUG::2014-09-25 17:09:05,578::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:09:05,874::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:05,874::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:05,874::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:05,875::__init__::232::IOProcess::(_processLogs) (107169) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:05,877::__init__::232::IOProcess::(_processLogs) (107169) Queuing response >Thread-81::DEBUG::2014-09-25 17:09:05,878::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:09:05,896::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00479751 s, 77.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:05,897::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:05,897::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:05,897::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:05,898::__init__::232::IOProcess::(_processLogs) (107170) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:05,902::__init__::232::IOProcess::(_processLogs) (107170) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:07,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:07,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207167 s, 49.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:08,596::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:08,598::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69680::DEBUG::2014-09-25 17:09:08,600::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:09,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:09,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193012 s, 53.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:10,279::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:10,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:10,280::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:10,280::__init__::232::IOProcess::(_processLogs) (107171) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:10,289::__init__::232::IOProcess::(_processLogs) (107171) Queuing response >Thread-79::DEBUG::2014-09-25 17:09:10,290::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:09:10,310::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00425526 s, 132 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:10,311::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:10,312::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:10,312::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:10,312::__init__::232::IOProcess::(_processLogs) (107172) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:10,314::__init__::232::IOProcess::(_processLogs) (107172) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:10,315::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:10,316::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:10,317::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:10,317::__init__::232::IOProcess::(_processLogs) (107173) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:10,317::__init__::232::IOProcess::(_processLogs) (107173) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:10,318::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:10,318::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:10,319::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:10,319::__init__::232::IOProcess::(_processLogs) (107174) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:10,319::__init__::232::IOProcess::(_processLogs) (107174) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:10,320::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:10,320::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:10,321::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:10,321::__init__::232::IOProcess::(_processLogs) (107175) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:10,322::__init__::232::IOProcess::(_processLogs) (107175) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:11,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:11,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210122 s, 48.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:11,620::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:11,621::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69681::DEBUG::2014-09-25 17:09:11,623::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-80::DEBUG::2014-09-25 17:09:12,022::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:09:12,026::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:12,027::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:12,027::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:12,027::__init__::232::IOProcess::(_processLogs) (107176) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:09:12,028::__init__::232::IOProcess::(_processLogs) (107176) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:12,029::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:12,029::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:12,030::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:12,030::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:12,030::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:12,030::__init__::232::IOProcess::(_processLogs) (107177) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:09:12,031::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:12,031::__init__::232::IOProcess::(_processLogs) (107178) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:09:12,031::__init__::232::IOProcess::(_processLogs) (107177) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:12,034::__init__::232::IOProcess::(_processLogs) (107178) Queuing response >Thread-80::DEBUG::2014-09-25 17:09:12,038::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:09:12,039::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:12,039::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:12,039::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:12,040::__init__::232::IOProcess::(_processLogs) (107179) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:09:12,043::__init__::232::IOProcess::(_processLogs) (107179) Queuing response >Thread-80::DEBUG::2014-09-25 17:09:12,043::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:09:12,044::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:12,044::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:12,044::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:12,045::__init__::232::IOProcess::(_processLogs) (107180) Got request for method 'readfile' >Thread-80::DEBUG::2014-09-25 17:09:12,045::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:09:12,045::__init__::232::IOProcess::(_processLogs) (107180) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:12,047::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:12,048::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:12,048::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:12,049::__init__::232::IOProcess::(_processLogs) (107181) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:12,049::__init__::232::IOProcess::(_processLogs) (107181) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:12,049::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:12,050::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:12,050::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:12,050::__init__::232::IOProcess::(_processLogs) (107182) Got request for method 'glob' >Thread-80::DEBUG::2014-09-25 17:09:12,051::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:09:12,051::__init__::232::IOProcess::(_processLogs) (107182) Queuing response >Thread-80::INFO::2014-09-25 17:09:12,052::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:09:12,052::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:09:12,053::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:12,053::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:12,053::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:12,054::__init__::232::IOProcess::(_processLogs) (107183) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:12,054::__init__::232::IOProcess::(_processLogs) (107183) Queuing response >Thread-80::DEBUG::2014-09-25 17:09:12,055::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:09:12,064::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000509519 s, 742 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:12,065::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:12,066::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:12,066::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:12,066::__init__::232::IOProcess::(_processLogs) (107184) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:12,067::__init__::232::IOProcess::(_processLogs) (107184) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:13,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:13,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218866 s, 46.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:14,421::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69685::DEBUG::2014-09-25 17:09:14,422::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:09:14,422::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69685::DEBUG::2014-09-25 17:09:14,425::task::595::Storage.TaskManager.Task::(_updateState) Task=`ce3838a6-48b6-4e0a-918f-fb04c992ec8a`::moving from state init -> state preparing >Thread-69685::INFO::2014-09-25 17:09:14,425::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69685::INFO::2014-09-25 17:09:14,426::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69685::DEBUG::2014-09-25 17:09:14,426::task::1191::Storage.TaskManager.Task::(prepare) Task=`ce3838a6-48b6-4e0a-918f-fb04c992ec8a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69685::DEBUG::2014-09-25 17:09:14,426::task::595::Storage.TaskManager.Task::(_updateState) Task=`ce3838a6-48b6-4e0a-918f-fb04c992ec8a`::moving from state preparing -> state finished >Thread-69685::DEBUG::2014-09-25 17:09:14,427::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69685::DEBUG::2014-09-25 17:09:14,427::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69685::DEBUG::2014-09-25 17:09:14,427::task::993::Storage.TaskManager.Task::(_decref) Task=`ce3838a6-48b6-4e0a-918f-fb04c992ec8a`::ref 0 aborting False >Thread-69685::DEBUG::2014-09-25 17:09:14,427::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69685::DEBUG::2014-09-25 17:09:14,428::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:14,442::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69686::DEBUG::2014-09-25 17:09:14,443::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:09:14,443::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69686::DEBUG::2014-09-25 17:09:14,445::task::595::Storage.TaskManager.Task::(_updateState) Task=`e6f630a5-11cb-405b-8dd5-3a869c902ccf`::moving from state init -> state preparing >Thread-69686::INFO::2014-09-25 17:09:14,445::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69686::DEBUG::2014-09-25 17:09:14,446::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8fb96a27-ef15-42ae-8921-8cc9efd3121f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69686::DEBUG::2014-09-25 17:09:14,446::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69686::DEBUG::2014-09-25 17:09:14,446::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69686::DEBUG::2014-09-25 17:09:14,447::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8fb96a27-ef15-42ae-8921-8cc9efd3121f`::Granted request >Thread-69686::DEBUG::2014-09-25 17:09:14,447::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e6f630a5-11cb-405b-8dd5-3a869c902ccf`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69686::DEBUG::2014-09-25 17:09:14,447::task::993::Storage.TaskManager.Task::(_decref) Task=`e6f630a5-11cb-405b-8dd5-3a869c902ccf`::ref 1 aborting False >Thread-69686::INFO::2014-09-25 17:09:14,452::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69686::DEBUG::2014-09-25 17:09:14,452::task::1191::Storage.TaskManager.Task::(prepare) Task=`e6f630a5-11cb-405b-8dd5-3a869c902ccf`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69686::DEBUG::2014-09-25 17:09:14,453::task::595::Storage.TaskManager.Task::(_updateState) Task=`e6f630a5-11cb-405b-8dd5-3a869c902ccf`::moving from state preparing -> state finished >Thread-69686::DEBUG::2014-09-25 17:09:14,453::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69686::DEBUG::2014-09-25 17:09:14,453::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69686::DEBUG::2014-09-25 17:09:14,454::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69686::DEBUG::2014-09-25 17:09:14,454::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69686::DEBUG::2014-09-25 17:09:14,454::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69686::DEBUG::2014-09-25 17:09:14,454::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69686::DEBUG::2014-09-25 17:09:14,455::task::993::Storage.TaskManager.Task::(_decref) Task=`e6f630a5-11cb-405b-8dd5-3a869c902ccf`::ref 0 aborting False >Thread-69686::DEBUG::2014-09-25 17:09:14,455::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69686::DEBUG::2014-09-25 17:09:14,456::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:14,645::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:14,646::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69687::DEBUG::2014-09-25 17:09:14,648::task::595::Storage.TaskManager.Task::(_updateState) Task=`9bfbea30-c1ad-40d5-838f-6e39ed16e0ed`::moving from state init -> state preparing >Thread-69687::INFO::2014-09-25 17:09:14,648::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69687::INFO::2014-09-25 17:09:14,649::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00425526', 'lastCheck': '4.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000509519', 'lastCheck': '2.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00479751', 'lastCheck': '8.7', 'valid': True}} >Thread-69687::DEBUG::2014-09-25 17:09:14,649::task::1191::Storage.TaskManager.Task::(prepare) Task=`9bfbea30-c1ad-40d5-838f-6e39ed16e0ed`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00425526', 'lastCheck': '4.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000509519', 'lastCheck': '2.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00479751', 'lastCheck': '8.7', 'valid': True}} >Thread-69687::DEBUG::2014-09-25 17:09:14,649::task::595::Storage.TaskManager.Task::(_updateState) Task=`9bfbea30-c1ad-40d5-838f-6e39ed16e0ed`::moving from state preparing -> state finished >Thread-69687::DEBUG::2014-09-25 17:09:14,649::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69687::DEBUG::2014-09-25 17:09:14,650::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69687::DEBUG::2014-09-25 17:09:14,650::task::993::Storage.TaskManager.Task::(_decref) Task=`9bfbea30-c1ad-40d5-838f-6e39ed16e0ed`::ref 0 aborting False >Thread-69687::DEBUG::2014-09-25 17:09:14,652::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:14,673::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:14,674::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69688::DEBUG::2014-09-25 17:09:14,676::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:15,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:15,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0224138 s, 45.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:15,903::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:15,904::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:15,904::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:15,904::__init__::232::IOProcess::(_processLogs) (107185) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:15,910::__init__::232::IOProcess::(_processLogs) (107185) Queuing response >Thread-81::DEBUG::2014-09-25 17:09:15,911::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:09:15,934::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00548007 s, 67.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:15,936::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:15,936::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:15,936::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:15,937::__init__::232::IOProcess::(_processLogs) (107186) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:15,941::__init__::232::IOProcess::(_processLogs) (107186) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:17,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:17,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172187 s, 59.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:17,729::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:17,731::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69689::DEBUG::2014-09-25 17:09:17,733::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:19,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:19,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161167 s, 63.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:20,322::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:20,322::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:20,323::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:20,323::__init__::232::IOProcess::(_processLogs) (107187) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:20,325::__init__::232::IOProcess::(_processLogs) (107187) Queuing response >Thread-79::DEBUG::2014-09-25 17:09:20,326::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:09:20,335::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000408725 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:20,336::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:20,337::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:20,337::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:20,338::__init__::232::IOProcess::(_processLogs) (107188) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:20,339::__init__::232::IOProcess::(_processLogs) (107188) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:20,339::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:20,339::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:20,340::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:20,341::__init__::232::IOProcess::(_processLogs) (107189) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:20,341::__init__::232::IOProcess::(_processLogs) (107189) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:20,341::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:20,342::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:20,342::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:20,343::__init__::232::IOProcess::(_processLogs) (107190) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:20,343::__init__::232::IOProcess::(_processLogs) (107190) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:20,343::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:20,344::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:20,344::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:20,345::__init__::232::IOProcess::(_processLogs) (107191) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:20,345::__init__::232::IOProcess::(_processLogs) (107191) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:20,746::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:20,747::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69690::DEBUG::2014-09-25 17:09:20,749::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:21,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:21,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181137 s, 56.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:22,069::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:22,070::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:22,070::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:22,070::__init__::232::IOProcess::(_processLogs) (107192) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:22,070::__init__::232::IOProcess::(_processLogs) (107192) Queuing response >Thread-80::DEBUG::2014-09-25 17:09:22,072::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:09:22,081::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000599692 s, 630 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:22,082::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:22,082::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:22,082::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:22,083::__init__::232::IOProcess::(_processLogs) (107193) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:22,083::__init__::232::IOProcess::(_processLogs) (107193) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:23,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:23,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155736 s, 65.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:23,761::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:23,763::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69691::DEBUG::2014-09-25 17:09:23,765::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:24,488::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69692::DEBUG::2014-09-25 17:09:24,489::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:09:24,489::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69692::DEBUG::2014-09-25 17:09:24,492::task::595::Storage.TaskManager.Task::(_updateState) Task=`d50458cc-83b1-41e1-bbcf-8ca5d0671029`::moving from state init -> state preparing >Thread-69692::INFO::2014-09-25 17:09:24,492::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69692::INFO::2014-09-25 17:09:24,492::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69692::DEBUG::2014-09-25 17:09:24,493::task::1191::Storage.TaskManager.Task::(prepare) Task=`d50458cc-83b1-41e1-bbcf-8ca5d0671029`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69692::DEBUG::2014-09-25 17:09:24,493::task::595::Storage.TaskManager.Task::(_updateState) Task=`d50458cc-83b1-41e1-bbcf-8ca5d0671029`::moving from state preparing -> state finished >Thread-69692::DEBUG::2014-09-25 17:09:24,493::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69692::DEBUG::2014-09-25 17:09:24,493::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69692::DEBUG::2014-09-25 17:09:24,494::task::993::Storage.TaskManager.Task::(_decref) Task=`d50458cc-83b1-41e1-bbcf-8ca5d0671029`::ref 0 aborting False >Thread-69692::DEBUG::2014-09-25 17:09:24,494::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69692::DEBUG::2014-09-25 17:09:24,494::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:24,508::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69693::DEBUG::2014-09-25 17:09:24,509::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:09:24,509::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69693::DEBUG::2014-09-25 17:09:24,511::task::595::Storage.TaskManager.Task::(_updateState) Task=`9d748028-3744-424f-b74e-11aa722a88ea`::moving from state init -> state preparing >Thread-69693::INFO::2014-09-25 17:09:24,512::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69693::DEBUG::2014-09-25 17:09:24,512::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aed7d019-b563-4f9e-b852-f7786ffa4aef`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69693::DEBUG::2014-09-25 17:09:24,513::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69693::DEBUG::2014-09-25 17:09:24,513::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69693::DEBUG::2014-09-25 17:09:24,513::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aed7d019-b563-4f9e-b852-f7786ffa4aef`::Granted request >Thread-69693::DEBUG::2014-09-25 17:09:24,514::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9d748028-3744-424f-b74e-11aa722a88ea`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69693::DEBUG::2014-09-25 17:09:24,514::task::993::Storage.TaskManager.Task::(_decref) Task=`9d748028-3744-424f-b74e-11aa722a88ea`::ref 1 aborting False >Thread-69693::INFO::2014-09-25 17:09:24,518::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69693::DEBUG::2014-09-25 17:09:24,519::task::1191::Storage.TaskManager.Task::(prepare) Task=`9d748028-3744-424f-b74e-11aa722a88ea`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69693::DEBUG::2014-09-25 17:09:24,519::task::595::Storage.TaskManager.Task::(_updateState) Task=`9d748028-3744-424f-b74e-11aa722a88ea`::moving from state preparing -> state finished >Thread-69693::DEBUG::2014-09-25 17:09:24,519::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69693::DEBUG::2014-09-25 17:09:24,519::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69693::DEBUG::2014-09-25 17:09:24,520::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69693::DEBUG::2014-09-25 17:09:24,520::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69693::DEBUG::2014-09-25 17:09:24,520::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69693::DEBUG::2014-09-25 17:09:24,521::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69693::DEBUG::2014-09-25 17:09:24,521::task::993::Storage.TaskManager.Task::(_decref) Task=`9d748028-3744-424f-b74e-11aa722a88ea`::ref 0 aborting False >Thread-69693::DEBUG::2014-09-25 17:09:24,521::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69693::DEBUG::2014-09-25 17:09:24,522::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:25,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:25,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209034 s, 49.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:25,943::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:25,944::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:25,944::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:25,944::__init__::232::IOProcess::(_processLogs) (107194) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:25,944::__init__::232::IOProcess::(_processLogs) (107194) Queuing response >Thread-81::DEBUG::2014-09-25 17:09:25,946::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:09:25,954::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000408459 s, 908 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:25,955::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:25,956::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:25,956::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:25,956::__init__::232::IOProcess::(_processLogs) (107195) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:25,957::__init__::232::IOProcess::(_processLogs) (107195) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:26,781::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:26,783::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69694::DEBUG::2014-09-25 17:09:26,784::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:27,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:27,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016242 s, 63.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:09:29,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:29,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0247603 s, 41.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:29,797::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:29,799::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69695::DEBUG::2014-09-25 17:09:29,802::task::595::Storage.TaskManager.Task::(_updateState) Task=`9dd4834d-894b-4afa-9a44-59252cb2dff6`::moving from state init -> state preparing >Thread-69695::INFO::2014-09-25 17:09:29,802::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69695::INFO::2014-09-25 17:09:29,803::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000408725', 'lastCheck': '9.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000599692', 'lastCheck': '7.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000408459', 'lastCheck': '3.8', 'valid': True}} >Thread-69695::DEBUG::2014-09-25 17:09:29,803::task::1191::Storage.TaskManager.Task::(prepare) Task=`9dd4834d-894b-4afa-9a44-59252cb2dff6`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000408725', 'lastCheck': '9.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000599692', 'lastCheck': '7.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000408459', 'lastCheck': '3.8', 'valid': True}} >Thread-69695::DEBUG::2014-09-25 17:09:29,803::task::595::Storage.TaskManager.Task::(_updateState) Task=`9dd4834d-894b-4afa-9a44-59252cb2dff6`::moving from state preparing -> state finished >Thread-69695::DEBUG::2014-09-25 17:09:29,803::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69695::DEBUG::2014-09-25 17:09:29,803::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69695::DEBUG::2014-09-25 17:09:29,804::task::993::Storage.TaskManager.Task::(_decref) Task=`9dd4834d-894b-4afa-9a44-59252cb2dff6`::ref 0 aborting False >Thread-69695::DEBUG::2014-09-25 17:09:29,806::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:29,830::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:29,832::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69696::DEBUG::2014-09-25 17:09:29,833::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:09:30,346::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:30,346::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:30,346::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:30,347::__init__::232::IOProcess::(_processLogs) (107196) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:30,348::__init__::232::IOProcess::(_processLogs) (107196) Queuing response >Thread-79::DEBUG::2014-09-25 17:09:30,349::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:09:30,362::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00340726 s, 165 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:30,363::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:30,364::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:30,364::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:30,364::__init__::232::IOProcess::(_processLogs) (107197) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:30,370::__init__::232::IOProcess::(_processLogs) (107197) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:30,371::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:30,371::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:30,372::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:30,372::__init__::232::IOProcess::(_processLogs) (107198) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:30,373::__init__::232::IOProcess::(_processLogs) (107198) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:30,373::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:30,374::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:30,374::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:30,375::__init__::232::IOProcess::(_processLogs) (107199) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:30,375::__init__::232::IOProcess::(_processLogs) (107199) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:30,375::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:30,375::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:30,376::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:30,377::__init__::232::IOProcess::(_processLogs) (107200) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:30,377::__init__::232::IOProcess::(_processLogs) (107200) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:31,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:31,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147912 s, 69.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:32,086::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:32,086::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:32,086::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:32,088::__init__::232::IOProcess::(_processLogs) (107201) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:09:32,088::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:09:32,089::__init__::232::IOProcess::(_processLogs) (107201) Queuing response >Thread-80::DEBUG::2014-09-25 17:09:32,098::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000776038 s, 487 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:32,099::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:32,100::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:32,100::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:32,102::__init__::232::IOProcess::(_processLogs) (107202) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:32,102::__init__::232::IOProcess::(_processLogs) (107202) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:32,894::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:32,896::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69697::DEBUG::2014-09-25 17:09:32,898::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:33,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:33,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163586 s, 62.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:34,551::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69698::DEBUG::2014-09-25 17:09:34,552::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:09:34,552::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69698::DEBUG::2014-09-25 17:09:34,555::task::595::Storage.TaskManager.Task::(_updateState) Task=`70e8d822-cce0-41ce-bd47-10de4441fd82`::moving from state init -> state preparing >Thread-69698::INFO::2014-09-25 17:09:34,555::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69698::INFO::2014-09-25 17:09:34,556::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69698::DEBUG::2014-09-25 17:09:34,556::task::1191::Storage.TaskManager.Task::(prepare) Task=`70e8d822-cce0-41ce-bd47-10de4441fd82`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69698::DEBUG::2014-09-25 17:09:34,556::task::595::Storage.TaskManager.Task::(_updateState) Task=`70e8d822-cce0-41ce-bd47-10de4441fd82`::moving from state preparing -> state finished >Thread-69698::DEBUG::2014-09-25 17:09:34,557::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69698::DEBUG::2014-09-25 17:09:34,557::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69698::DEBUG::2014-09-25 17:09:34,557::task::993::Storage.TaskManager.Task::(_decref) Task=`70e8d822-cce0-41ce-bd47-10de4441fd82`::ref 0 aborting False >Thread-69698::DEBUG::2014-09-25 17:09:34,557::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69698::DEBUG::2014-09-25 17:09:34,558::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:34,568::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69699::DEBUG::2014-09-25 17:09:34,570::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:09:34,570::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69699::DEBUG::2014-09-25 17:09:34,572::task::595::Storage.TaskManager.Task::(_updateState) Task=`da8e4b5f-9677-4d46-8ff8-1e4a74c55979`::moving from state init -> state preparing >Thread-69699::INFO::2014-09-25 17:09:34,572::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69699::DEBUG::2014-09-25 17:09:34,573::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2e335f98-deab-4d3c-abbc-f9c5f9ba19a9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69699::DEBUG::2014-09-25 17:09:34,573::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69699::DEBUG::2014-09-25 17:09:34,573::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69699::DEBUG::2014-09-25 17:09:34,574::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2e335f98-deab-4d3c-abbc-f9c5f9ba19a9`::Granted request >Thread-69699::DEBUG::2014-09-25 17:09:34,574::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`da8e4b5f-9677-4d46-8ff8-1e4a74c55979`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69699::DEBUG::2014-09-25 17:09:34,574::task::993::Storage.TaskManager.Task::(_decref) Task=`da8e4b5f-9677-4d46-8ff8-1e4a74c55979`::ref 1 aborting False >Thread-69699::INFO::2014-09-25 17:09:34,579::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69699::DEBUG::2014-09-25 17:09:34,579::task::1191::Storage.TaskManager.Task::(prepare) Task=`da8e4b5f-9677-4d46-8ff8-1e4a74c55979`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69699::DEBUG::2014-09-25 17:09:34,579::task::595::Storage.TaskManager.Task::(_updateState) Task=`da8e4b5f-9677-4d46-8ff8-1e4a74c55979`::moving from state preparing -> state finished >Thread-69699::DEBUG::2014-09-25 17:09:34,580::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69699::DEBUG::2014-09-25 17:09:34,580::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69699::DEBUG::2014-09-25 17:09:34,580::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69699::DEBUG::2014-09-25 17:09:34,581::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69699::DEBUG::2014-09-25 17:09:34,581::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69699::DEBUG::2014-09-25 17:09:34,581::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69699::DEBUG::2014-09-25 17:09:34,581::task::993::Storage.TaskManager.Task::(_decref) Task=`da8e4b5f-9677-4d46-8ff8-1e4a74c55979`::ref 0 aborting False >Thread-69699::DEBUG::2014-09-25 17:09:34,582::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69699::DEBUG::2014-09-25 17:09:34,582::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:35,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:35,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158811 s, 64.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:35,916::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:35,917::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69700::DEBUG::2014-09-25 17:09:35,920::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:09:35,959::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:35,959::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:35,959::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:35,960::__init__::232::IOProcess::(_processLogs) (107203) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:35,966::__init__::232::IOProcess::(_processLogs) (107203) Queuing response >Thread-81::DEBUG::2014-09-25 17:09:35,967::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:09:35,984::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00666101 s, 55.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:35,986::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:35,986::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:35,986::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:35,986::__init__::232::IOProcess::(_processLogs) (107204) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:35,993::__init__::232::IOProcess::(_processLogs) (107204) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:37,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:37,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132794 s, 77.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:38,933::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:38,935::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69701::DEBUG::2014-09-25 17:09:38,937::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:39,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:39,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215787 s, 47.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:40,378::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:40,378::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:40,378::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:40,379::__init__::232::IOProcess::(_processLogs) (107205) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:40,382::__init__::232::IOProcess::(_processLogs) (107205) Queuing response >Thread-79::DEBUG::2014-09-25 17:09:40,383::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:09:40,402::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00272441 s, 206 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:40,403::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:40,404::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:40,404::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:40,404::__init__::232::IOProcess::(_processLogs) (107206) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:40,411::__init__::232::IOProcess::(_processLogs) (107206) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:40,412::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:40,412::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:40,413::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:40,413::__init__::232::IOProcess::(_processLogs) (107207) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:40,420::__init__::232::IOProcess::(_processLogs) (107207) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:40,421::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:40,421::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:40,422::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:40,422::__init__::232::IOProcess::(_processLogs) (107208) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:40,433::__init__::232::IOProcess::(_processLogs) (107208) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:40,434::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:40,434::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:40,435::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:40,435::__init__::232::IOProcess::(_processLogs) (107209) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:40,441::__init__::232::IOProcess::(_processLogs) (107209) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:41,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:41,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156309 s, 65.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:41,951::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:41,954::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69702::DEBUG::2014-09-25 17:09:41,958::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:09:42,102::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:42,103::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:42,103::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:42,103::__init__::232::IOProcess::(_processLogs) (107210) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:42,103::__init__::232::IOProcess::(_processLogs) (107210) Queuing response >Thread-80::DEBUG::2014-09-25 17:09:42,104::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:09:42,113::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000412031 s, 917 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:42,114::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:42,115::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:42,115::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:42,115::__init__::232::IOProcess::(_processLogs) (107211) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:42,115::__init__::232::IOProcess::(_processLogs) (107211) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:43,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:44,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167087 s, 61.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:44,607::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69703::DEBUG::2014-09-25 17:09:44,609::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:09:44,609::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69703::DEBUG::2014-09-25 17:09:44,611::task::595::Storage.TaskManager.Task::(_updateState) Task=`53039cf5-9ac2-432f-862d-1a2bd9a21efd`::moving from state init -> state preparing >Thread-69703::INFO::2014-09-25 17:09:44,612::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69703::INFO::2014-09-25 17:09:44,612::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69703::DEBUG::2014-09-25 17:09:44,613::task::1191::Storage.TaskManager.Task::(prepare) Task=`53039cf5-9ac2-432f-862d-1a2bd9a21efd`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69703::DEBUG::2014-09-25 17:09:44,613::task::595::Storage.TaskManager.Task::(_updateState) Task=`53039cf5-9ac2-432f-862d-1a2bd9a21efd`::moving from state preparing -> state finished >Thread-69703::DEBUG::2014-09-25 17:09:44,613::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69703::DEBUG::2014-09-25 17:09:44,613::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69703::DEBUG::2014-09-25 17:09:44,614::task::993::Storage.TaskManager.Task::(_decref) Task=`53039cf5-9ac2-432f-862d-1a2bd9a21efd`::ref 0 aborting False >Thread-69703::DEBUG::2014-09-25 17:09:44,614::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69703::DEBUG::2014-09-25 17:09:44,614::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:44,626::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:44,627::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69704::DEBUG::2014-09-25 17:09:44,627::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69704::DEBUG::2014-09-25 17:09:44,629::task::595::Storage.TaskManager.Task::(_updateState) Task=`f58d41e9-2615-42ac-bb3b-59fcb8cc41aa`::moving from state init -> state preparing >Thread-69704::INFO::2014-09-25 17:09:44,630::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69704::DEBUG::2014-09-25 17:09:44,630::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8c58eb18-b02d-4e68-88dd-cf2fb8739052`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69704::DEBUG::2014-09-25 17:09:44,631::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69704::DEBUG::2014-09-25 17:09:44,631::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69704::DEBUG::2014-09-25 17:09:44,631::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8c58eb18-b02d-4e68-88dd-cf2fb8739052`::Granted request >Thread-69704::DEBUG::2014-09-25 17:09:44,632::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f58d41e9-2615-42ac-bb3b-59fcb8cc41aa`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69704::DEBUG::2014-09-25 17:09:44,632::task::993::Storage.TaskManager.Task::(_decref) Task=`f58d41e9-2615-42ac-bb3b-59fcb8cc41aa`::ref 1 aborting False >Thread-69704::INFO::2014-09-25 17:09:44,637::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69704::DEBUG::2014-09-25 17:09:44,637::task::1191::Storage.TaskManager.Task::(prepare) Task=`f58d41e9-2615-42ac-bb3b-59fcb8cc41aa`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69704::DEBUG::2014-09-25 17:09:44,637::task::595::Storage.TaskManager.Task::(_updateState) Task=`f58d41e9-2615-42ac-bb3b-59fcb8cc41aa`::moving from state preparing -> state finished >Thread-69704::DEBUG::2014-09-25 17:09:44,638::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69704::DEBUG::2014-09-25 17:09:44,638::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69704::DEBUG::2014-09-25 17:09:44,638::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69704::DEBUG::2014-09-25 17:09:44,638::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69704::DEBUG::2014-09-25 17:09:44,639::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69704::DEBUG::2014-09-25 17:09:44,639::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69704::DEBUG::2014-09-25 17:09:44,639::task::993::Storage.TaskManager.Task::(_decref) Task=`f58d41e9-2615-42ac-bb3b-59fcb8cc41aa`::ref 0 aborting False >Thread-69704::DEBUG::2014-09-25 17:09:44,640::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69704::DEBUG::2014-09-25 17:09:44,640::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:44,975::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:44,977::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69705::DEBUG::2014-09-25 17:09:44,979::task::595::Storage.TaskManager.Task::(_updateState) Task=`f56533fe-451f-4138-926a-49bd0ac06738`::moving from state init -> state preparing >Thread-69705::INFO::2014-09-25 17:09:44,980::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69705::INFO::2014-09-25 17:09:44,980::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00272441', 'lastCheck': '4.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000412031', 'lastCheck': '2.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00666101', 'lastCheck': '9.0', 'valid': True}} >Thread-69705::DEBUG::2014-09-25 17:09:44,980::task::1191::Storage.TaskManager.Task::(prepare) Task=`f56533fe-451f-4138-926a-49bd0ac06738`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00272441', 'lastCheck': '4.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000412031', 'lastCheck': '2.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00666101', 'lastCheck': '9.0', 'valid': True}} >Thread-69705::DEBUG::2014-09-25 17:09:44,981::task::595::Storage.TaskManager.Task::(_updateState) Task=`f56533fe-451f-4138-926a-49bd0ac06738`::moving from state preparing -> state finished >Thread-69705::DEBUG::2014-09-25 17:09:44,981::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69705::DEBUG::2014-09-25 17:09:44,981::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69705::DEBUG::2014-09-25 17:09:44,981::task::993::Storage.TaskManager.Task::(_decref) Task=`f56533fe-451f-4138-926a-49bd0ac06738`::ref 0 aborting False >Thread-69705::DEBUG::2014-09-25 17:09:44,984::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:45,007::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:45,008::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69706::DEBUG::2014-09-25 17:09:45,010::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:09:45,994::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:45,995::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:45,995::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:45,995::__init__::232::IOProcess::(_processLogs) (107212) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:45,998::__init__::232::IOProcess::(_processLogs) (107212) Queuing response >Thread-81::DEBUG::2014-09-25 17:09:45,999::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:09:46,014::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00278868 s, 133 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:46,017::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:46,018::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:46,019::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:46,021::__init__::232::IOProcess::(_processLogs) (107213) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:46,022::__init__::232::IOProcess::(_processLogs) (107213) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:46,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:46,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0242642 s, 42.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:48,079::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:48,081::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69707::DEBUG::2014-09-25 17:09:48,083::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:48,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:48,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133889 s, 76.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:09:50,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:50,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131124 s, 78.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:50,443::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:50,443::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:50,444::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:09:50,445::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:09:50,445::__init__::232::IOProcess::(_processLogs) (107214) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:50,453::__init__::232::IOProcess::(_processLogs) (107214) Queuing response >Thread-79::DEBUG::2014-09-25 17:09:50,455::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000334114 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:50,456::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:50,456::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:50,456::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:50,456::__init__::232::IOProcess::(_processLogs) (107215) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:50,457::__init__::232::IOProcess::(_processLogs) (107215) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:50,459::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:50,459::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:50,459::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:50,460::__init__::232::IOProcess::(_processLogs) (107216) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:50,460::__init__::232::IOProcess::(_processLogs) (107216) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:50,461::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:50,461::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:50,461::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:50,462::__init__::232::IOProcess::(_processLogs) (107217) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:50,462::__init__::232::IOProcess::(_processLogs) (107217) Queuing response >Thread-29::DEBUG::2014-09-25 17:09:50,463::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:50,463::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:50,464::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:50,464::__init__::232::IOProcess::(_processLogs) (107218) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:09:50,464::__init__::232::IOProcess::(_processLogs) (107218) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:51,098::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:51,100::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69708::DEBUG::2014-09-25 17:09:51,102::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:09:52,118::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:52,118::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:52,119::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:52,120::__init__::232::IOProcess::(_processLogs) (107219) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:09:52,120::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:09:52,122::__init__::232::IOProcess::(_processLogs) (107219) Queuing response >Thread-80::DEBUG::2014-09-25 17:09:52,130::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000469307 s, 805 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:52,131::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:52,132::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:52,132::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:52,133::__init__::232::IOProcess::(_processLogs) (107220) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:52,134::__init__::232::IOProcess::(_processLogs) (107220) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:52,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:52,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175581 s, 58.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:54,113::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:54,115::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69709::DEBUG::2014-09-25 17:09:54,117::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:54,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:54,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157734 s, 64.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:54,665::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:54,666::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69710::DEBUG::2014-09-25 17:09:54,666::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69710::DEBUG::2014-09-25 17:09:54,668::task::595::Storage.TaskManager.Task::(_updateState) Task=`530e5650-1c7c-4946-bb79-93172e316a82`::moving from state init -> state preparing >Thread-69710::INFO::2014-09-25 17:09:54,669::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69710::INFO::2014-09-25 17:09:54,669::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69710::DEBUG::2014-09-25 17:09:54,669::task::1191::Storage.TaskManager.Task::(prepare) Task=`530e5650-1c7c-4946-bb79-93172e316a82`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69710::DEBUG::2014-09-25 17:09:54,670::task::595::Storage.TaskManager.Task::(_updateState) Task=`530e5650-1c7c-4946-bb79-93172e316a82`::moving from state preparing -> state finished >Thread-69710::DEBUG::2014-09-25 17:09:54,670::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69710::DEBUG::2014-09-25 17:09:54,670::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69710::DEBUG::2014-09-25 17:09:54,670::task::993::Storage.TaskManager.Task::(_decref) Task=`530e5650-1c7c-4946-bb79-93172e316a82`::ref 0 aborting False >Thread-69710::DEBUG::2014-09-25 17:09:54,671::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69710::DEBUG::2014-09-25 17:09:54,671::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:54,685::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69711::DEBUG::2014-09-25 17:09:54,687::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:09:54,686::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69711::DEBUG::2014-09-25 17:09:54,689::task::595::Storage.TaskManager.Task::(_updateState) Task=`3286255b-341b-4506-8e27-69857896bde6`::moving from state init -> state preparing >Thread-69711::INFO::2014-09-25 17:09:54,689::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69711::DEBUG::2014-09-25 17:09:54,689::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e03ffefa-7970-4d1c-bb99-c3d4ab75baea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69711::DEBUG::2014-09-25 17:09:54,690::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69711::DEBUG::2014-09-25 17:09:54,690::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69711::DEBUG::2014-09-25 17:09:54,690::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e03ffefa-7970-4d1c-bb99-c3d4ab75baea`::Granted request >Thread-69711::DEBUG::2014-09-25 17:09:54,691::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3286255b-341b-4506-8e27-69857896bde6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69711::DEBUG::2014-09-25 17:09:54,691::task::993::Storage.TaskManager.Task::(_decref) Task=`3286255b-341b-4506-8e27-69857896bde6`::ref 1 aborting False >Thread-69711::INFO::2014-09-25 17:09:54,696::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69711::DEBUG::2014-09-25 17:09:54,696::task::1191::Storage.TaskManager.Task::(prepare) Task=`3286255b-341b-4506-8e27-69857896bde6`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69711::DEBUG::2014-09-25 17:09:54,696::task::595::Storage.TaskManager.Task::(_updateState) Task=`3286255b-341b-4506-8e27-69857896bde6`::moving from state preparing -> state finished >Thread-69711::DEBUG::2014-09-25 17:09:54,696::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69711::DEBUG::2014-09-25 17:09:54,697::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69711::DEBUG::2014-09-25 17:09:54,697::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69711::DEBUG::2014-09-25 17:09:54,697::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69711::DEBUG::2014-09-25 17:09:54,698::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69711::DEBUG::2014-09-25 17:09:54,698::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69711::DEBUG::2014-09-25 17:09:54,698::task::993::Storage.TaskManager.Task::(_decref) Task=`3286255b-341b-4506-8e27-69857896bde6`::ref 0 aborting False >Thread-69711::DEBUG::2014-09-25 17:09:54,699::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69711::DEBUG::2014-09-25 17:09:54,699::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:09:56,025::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:56,026::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:56,026::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:56,026::__init__::232::IOProcess::(_processLogs) (107221) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:56,036::__init__::232::IOProcess::(_processLogs) (107221) Queuing response >Thread-81::DEBUG::2014-09-25 17:09:56,037::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:09:56,061::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00410035 s, 90.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:09:56,063::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:09:56,063::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:09:56,063::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:09:56,063::__init__::232::IOProcess::(_processLogs) (107222) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:09:56,071::__init__::232::IOProcess::(_processLogs) (107222) Queuing response >Dummy-148::DEBUG::2014-09-25 17:09:56,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:56,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0366771 s, 27.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:09:57,130::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:09:57,132::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69712::DEBUG::2014-09-25 17:09:57,134::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:09:58,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:09:58,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.213317 s, 4.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:00,145::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:00,147::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69713::DEBUG::2014-09-25 17:10:00,149::task::595::Storage.TaskManager.Task::(_updateState) Task=`b16251e9-f52c-4782-bb42-d0702ea169dc`::moving from state init -> state preparing >Thread-69713::INFO::2014-09-25 17:10:00,149::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69713::INFO::2014-09-25 17:10:00,150::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000334114', 'lastCheck': '9.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000469307', 'lastCheck': '8.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00410035', 'lastCheck': '4.1', 'valid': True}} >Thread-69713::DEBUG::2014-09-25 17:10:00,150::task::1191::Storage.TaskManager.Task::(prepare) Task=`b16251e9-f52c-4782-bb42-d0702ea169dc`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000334114', 'lastCheck': '9.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000469307', 'lastCheck': '8.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00410035', 'lastCheck': '4.1', 'valid': True}} >Thread-69713::DEBUG::2014-09-25 17:10:00,150::task::595::Storage.TaskManager.Task::(_updateState) Task=`b16251e9-f52c-4782-bb42-d0702ea169dc`::moving from state preparing -> state finished >Thread-69713::DEBUG::2014-09-25 17:10:00,151::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69713::DEBUG::2014-09-25 17:10:00,151::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69713::DEBUG::2014-09-25 17:10:00,151::task::993::Storage.TaskManager.Task::(_decref) Task=`b16251e9-f52c-4782-bb42-d0702ea169dc`::ref 0 aborting False >Thread-69713::DEBUG::2014-09-25 17:10:00,153::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:00,168::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:00,169::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69714::DEBUG::2014-09-25 17:10:00,171::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:10:00,464::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:00,465::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:00,465::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:00,465::__init__::232::IOProcess::(_processLogs) (107223) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:00,467::__init__::232::IOProcess::(_processLogs) (107223) Queuing response >Thread-79::DEBUG::2014-09-25 17:10:00,469::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:10:00,490::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00426018 s, 132 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:00,491::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:00,492::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:00,492::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:00,492::__init__::232::IOProcess::(_processLogs) (107224) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:00,494::__init__::232::IOProcess::(_processLogs) (107224) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:00,495::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:00,496::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:00,496::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:00,496::__init__::232::IOProcess::(_processLogs) (107225) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:00,496::__init__::232::IOProcess::(_processLogs) (107225) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:00,497::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:00,498::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:00,498::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:00,499::__init__::232::IOProcess::(_processLogs) (107226) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:00,499::__init__::232::IOProcess::(_processLogs) (107226) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:00,499::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:00,500::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:00,500::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:00,501::__init__::232::IOProcess::(_processLogs) (107227) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:00,501::__init__::232::IOProcess::(_processLogs) (107227) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:00,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:00,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0190644 s, 53.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:02,135::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:02,135::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:02,136::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:02,137::__init__::232::IOProcess::(_processLogs) (107228) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:10:02,137::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:10:02,138::__init__::232::IOProcess::(_processLogs) (107228) Queuing response >Thread-80::DEBUG::2014-09-25 17:10:02,147::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.0004149 s, 911 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:02,148::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:02,148::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:02,149::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:02,150::__init__::232::IOProcess::(_processLogs) (107229) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:02,151::__init__::232::IOProcess::(_processLogs) (107229) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:02,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:02,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158661 s, 64.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:03,219::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:03,221::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69715::DEBUG::2014-09-25 17:10:03,223::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:04,733::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:04,734::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69716::DEBUG::2014-09-25 17:10:04,734::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69716::DEBUG::2014-09-25 17:10:04,735::task::595::Storage.TaskManager.Task::(_updateState) Task=`163ab2b7-b9a3-4313-8c44-807ecebe6f3f`::moving from state init -> state preparing >Thread-69716::INFO::2014-09-25 17:10:04,736::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69716::INFO::2014-09-25 17:10:04,736::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69716::DEBUG::2014-09-25 17:10:04,736::task::1191::Storage.TaskManager.Task::(prepare) Task=`163ab2b7-b9a3-4313-8c44-807ecebe6f3f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69716::DEBUG::2014-09-25 17:10:04,737::task::595::Storage.TaskManager.Task::(_updateState) Task=`163ab2b7-b9a3-4313-8c44-807ecebe6f3f`::moving from state preparing -> state finished >Thread-69716::DEBUG::2014-09-25 17:10:04,737::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69716::DEBUG::2014-09-25 17:10:04,737::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69716::DEBUG::2014-09-25 17:10:04,737::task::993::Storage.TaskManager.Task::(_decref) Task=`163ab2b7-b9a3-4313-8c44-807ecebe6f3f`::ref 0 aborting False >Thread-69716::DEBUG::2014-09-25 17:10:04,738::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69716::DEBUG::2014-09-25 17:10:04,738::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:04,750::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69717::DEBUG::2014-09-25 17:10:04,752::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:10:04,752::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69717::DEBUG::2014-09-25 17:10:04,754::task::595::Storage.TaskManager.Task::(_updateState) Task=`eef74acc-8dd1-44be-9972-be1d375bfb73`::moving from state init -> state preparing >Thread-69717::INFO::2014-09-25 17:10:04,754::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69717::DEBUG::2014-09-25 17:10:04,755::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2ce3193b-65c0-4722-b7bf-a84b2eb7f1f1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69717::DEBUG::2014-09-25 17:10:04,755::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69717::DEBUG::2014-09-25 17:10:04,755::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69717::DEBUG::2014-09-25 17:10:04,756::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2ce3193b-65c0-4722-b7bf-a84b2eb7f1f1`::Granted request >Thread-69717::DEBUG::2014-09-25 17:10:04,756::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`eef74acc-8dd1-44be-9972-be1d375bfb73`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69717::DEBUG::2014-09-25 17:10:04,757::task::993::Storage.TaskManager.Task::(_decref) Task=`eef74acc-8dd1-44be-9972-be1d375bfb73`::ref 1 aborting False >Thread-69717::INFO::2014-09-25 17:10:04,761::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69717::DEBUG::2014-09-25 17:10:04,761::task::1191::Storage.TaskManager.Task::(prepare) Task=`eef74acc-8dd1-44be-9972-be1d375bfb73`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69717::DEBUG::2014-09-25 17:10:04,762::task::595::Storage.TaskManager.Task::(_updateState) Task=`eef74acc-8dd1-44be-9972-be1d375bfb73`::moving from state preparing -> state finished >Thread-69717::DEBUG::2014-09-25 17:10:04,762::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69717::DEBUG::2014-09-25 17:10:04,762::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69717::DEBUG::2014-09-25 17:10:04,762::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69717::DEBUG::2014-09-25 17:10:04,763::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69717::DEBUG::2014-09-25 17:10:04,763::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69717::DEBUG::2014-09-25 17:10:04,763::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69717::DEBUG::2014-09-25 17:10:04,764::task::993::Storage.TaskManager.Task::(_decref) Task=`eef74acc-8dd1-44be-9972-be1d375bfb73`::ref 0 aborting False >Thread-69717::DEBUG::2014-09-25 17:10:04,764::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69717::DEBUG::2014-09-25 17:10:04,765::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:04,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:04,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181392 s, 56.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:06,073::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:06,073::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:06,074::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:06,074::__init__::232::IOProcess::(_processLogs) (107230) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:06,077::__init__::232::IOProcess::(_processLogs) (107230) Queuing response >Thread-81::DEBUG::2014-09-25 17:10:06,078::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:10:06,095::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00326981 s, 113 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:06,096::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:06,097::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:06,097::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:06,097::__init__::232::IOProcess::(_processLogs) (107231) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:06,105::__init__::232::IOProcess::(_processLogs) (107231) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:06,240::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:06,241::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69718::DEBUG::2014-09-25 17:10:06,243::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:06,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:06,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.02503 s, 40.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:10:08,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:08,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132039 s, 77.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:09,267::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:09,268::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69719::DEBUG::2014-09-25 17:10:09,270::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:10:10,502::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:10,502::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:10,502::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:10,503::__init__::232::IOProcess::(_processLogs) (107232) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:10,510::__init__::232::IOProcess::(_processLogs) (107232) Queuing response >Thread-79::DEBUG::2014-09-25 17:10:10,511::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:10:10,531::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00479276 s, 117 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:10,532::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:10,533::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:10,533::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:10,533::__init__::232::IOProcess::(_processLogs) (107233) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:10,535::__init__::232::IOProcess::(_processLogs) (107233) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:10,537::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:10,537::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:10,538::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:10,538::__init__::232::IOProcess::(_processLogs) (107234) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:10,538::__init__::232::IOProcess::(_processLogs) (107234) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:10,539::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:10,539::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:10,539::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:10,539::__init__::232::IOProcess::(_processLogs) (107235) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:10,540::__init__::232::IOProcess::(_processLogs) (107235) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:10,540::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:10,541::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:10,541::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:10,541::__init__::232::IOProcess::(_processLogs) (107236) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:10,541::__init__::232::IOProcess::(_processLogs) (107236) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:11,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:11,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173512 s, 59.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:12,152::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:12,152::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:12,152::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:12,153::__init__::232::IOProcess::(_processLogs) (107237) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:12,153::__init__::232::IOProcess::(_processLogs) (107237) Queuing response >Thread-80::DEBUG::2014-09-25 17:10:12,154::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:10:12,163::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000379973 s, 995 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:12,164::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:12,164::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:12,165::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:12,165::__init__::232::IOProcess::(_processLogs) (107238) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:12,166::__init__::232::IOProcess::(_processLogs) (107238) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:12,282::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:12,283::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69720::DEBUG::2014-09-25 17:10:12,285::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:13,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:13,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014847 s, 69.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:14,790::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:14,791::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69721::DEBUG::2014-09-25 17:10:14,791::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69721::DEBUG::2014-09-25 17:10:14,794::task::595::Storage.TaskManager.Task::(_updateState) Task=`b92e75b7-3141-44b2-ab22-e612e3a3e36b`::moving from state init -> state preparing >Thread-69721::INFO::2014-09-25 17:10:14,794::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69721::INFO::2014-09-25 17:10:14,795::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69721::DEBUG::2014-09-25 17:10:14,795::task::1191::Storage.TaskManager.Task::(prepare) Task=`b92e75b7-3141-44b2-ab22-e612e3a3e36b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69721::DEBUG::2014-09-25 17:10:14,795::task::595::Storage.TaskManager.Task::(_updateState) Task=`b92e75b7-3141-44b2-ab22-e612e3a3e36b`::moving from state preparing -> state finished >Thread-69721::DEBUG::2014-09-25 17:10:14,795::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69721::DEBUG::2014-09-25 17:10:14,795::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69721::DEBUG::2014-09-25 17:10:14,796::task::993::Storage.TaskManager.Task::(_decref) Task=`b92e75b7-3141-44b2-ab22-e612e3a3e36b`::ref 0 aborting False >Thread-69721::DEBUG::2014-09-25 17:10:14,796::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69721::DEBUG::2014-09-25 17:10:14,797::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:14,807::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:14,809::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69722::DEBUG::2014-09-25 17:10:14,809::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69722::DEBUG::2014-09-25 17:10:14,811::task::595::Storage.TaskManager.Task::(_updateState) Task=`fd9198f8-7692-4b7e-bd69-2c1976a7e77f`::moving from state init -> state preparing >Thread-69722::INFO::2014-09-25 17:10:14,811::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69722::DEBUG::2014-09-25 17:10:14,812::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5697d91c-e451-4abf-8bc7-4cc325c1259a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69722::DEBUG::2014-09-25 17:10:14,812::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69722::DEBUG::2014-09-25 17:10:14,812::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69722::DEBUG::2014-09-25 17:10:14,813::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5697d91c-e451-4abf-8bc7-4cc325c1259a`::Granted request >Thread-69722::DEBUG::2014-09-25 17:10:14,813::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`fd9198f8-7692-4b7e-bd69-2c1976a7e77f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69722::DEBUG::2014-09-25 17:10:14,814::task::993::Storage.TaskManager.Task::(_decref) Task=`fd9198f8-7692-4b7e-bd69-2c1976a7e77f`::ref 1 aborting False >Thread-69722::INFO::2014-09-25 17:10:14,818::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69722::DEBUG::2014-09-25 17:10:14,818::task::1191::Storage.TaskManager.Task::(prepare) Task=`fd9198f8-7692-4b7e-bd69-2c1976a7e77f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69722::DEBUG::2014-09-25 17:10:14,819::task::595::Storage.TaskManager.Task::(_updateState) Task=`fd9198f8-7692-4b7e-bd69-2c1976a7e77f`::moving from state preparing -> state finished >Thread-69722::DEBUG::2014-09-25 17:10:14,819::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69722::DEBUG::2014-09-25 17:10:14,819::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69722::DEBUG::2014-09-25 17:10:14,820::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69722::DEBUG::2014-09-25 17:10:14,820::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69722::DEBUG::2014-09-25 17:10:14,820::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69722::DEBUG::2014-09-25 17:10:14,820::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69722::DEBUG::2014-09-25 17:10:14,821::task::993::Storage.TaskManager.Task::(_decref) Task=`fd9198f8-7692-4b7e-bd69-2c1976a7e77f`::ref 0 aborting False >Thread-69722::DEBUG::2014-09-25 17:10:14,821::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69722::DEBUG::2014-09-25 17:10:14,822::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:15,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:15,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192913 s, 53.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:15,296::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:15,298::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69723::DEBUG::2014-09-25 17:10:15,301::task::595::Storage.TaskManager.Task::(_updateState) Task=`32a5f22d-fd07-4d11-94ce-726db8108a34`::moving from state init -> state preparing >Thread-69723::INFO::2014-09-25 17:10:15,301::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69723::INFO::2014-09-25 17:10:15,301::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00479276', 'lastCheck': '4.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000379973', 'lastCheck': '3.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00326981', 'lastCheck': '9.2', 'valid': True}} >Thread-69723::DEBUG::2014-09-25 17:10:15,302::task::1191::Storage.TaskManager.Task::(prepare) Task=`32a5f22d-fd07-4d11-94ce-726db8108a34`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00479276', 'lastCheck': '4.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000379973', 'lastCheck': '3.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00326981', 'lastCheck': '9.2', 'valid': True}} >Thread-69723::DEBUG::2014-09-25 17:10:15,302::task::595::Storage.TaskManager.Task::(_updateState) Task=`32a5f22d-fd07-4d11-94ce-726db8108a34`::moving from state preparing -> state finished >Thread-69723::DEBUG::2014-09-25 17:10:15,302::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69723::DEBUG::2014-09-25 17:10:15,302::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69723::DEBUG::2014-09-25 17:10:15,303::task::993::Storage.TaskManager.Task::(_decref) Task=`32a5f22d-fd07-4d11-94ce-726db8108a34`::ref 0 aborting False >Thread-69723::DEBUG::2014-09-25 17:10:15,305::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:15,321::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:15,322::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69724::DEBUG::2014-09-25 17:10:15,325::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:10:16,108::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:16,110::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-81::DEBUG::2014-09-25 17:10:16,112::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:10:16,112::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:16,122::__init__::232::IOProcess::(_processLogs) (107239) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:16,124::__init__::232::IOProcess::(_processLogs) (107239) Queuing response >Thread-81::DEBUG::2014-09-25 17:10:16,125::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000336427 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:16,129::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:16,130::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:16,131::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:16,132::__init__::232::IOProcess::(_processLogs) (107240) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:16,134::__init__::232::IOProcess::(_processLogs) (107240) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:17,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:17,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132046 s, 77.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:18,368::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:18,370::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69725::DEBUG::2014-09-25 17:10:18,371::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:19,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:19,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132598 s, 77.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:20,543::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:20,543::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:20,544::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:20,544::__init__::232::IOProcess::(_processLogs) (107241) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:20,544::__init__::232::IOProcess::(_processLogs) (107241) Queuing response >Thread-79::DEBUG::2014-09-25 17:10:20,546::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:10:20,566::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00875 s, 64.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:20,567::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:20,567::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:20,568::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:20,568::__init__::232::IOProcess::(_processLogs) (107242) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:20,571::__init__::232::IOProcess::(_processLogs) (107242) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:20,572::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:20,573::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:20,573::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:20,574::__init__::232::IOProcess::(_processLogs) (107243) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:20,574::__init__::232::IOProcess::(_processLogs) (107243) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:20,575::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:20,575::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:20,575::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:20,576::__init__::232::IOProcess::(_processLogs) (107244) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:20,576::__init__::232::IOProcess::(_processLogs) (107244) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:20,577::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:20,577::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:20,578::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:20,579::__init__::232::IOProcess::(_processLogs) (107245) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:20,579::__init__::232::IOProcess::(_processLogs) (107245) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:21,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:21,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153058 s, 66.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:21,385::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:21,387::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69726::DEBUG::2014-09-25 17:10:21,389::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:10:22,168::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:22,168::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:22,169::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:22,169::__init__::232::IOProcess::(_processLogs) (107246) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:22,169::__init__::232::IOProcess::(_processLogs) (107246) Queuing response >Thread-80::DEBUG::2014-09-25 17:10:22,171::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:10:22,179::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000519427 s, 728 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:22,181::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:22,181::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:22,181::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:22,181::__init__::232::IOProcess::(_processLogs) (107247) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:22,182::__init__::232::IOProcess::(_processLogs) (107247) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:23,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:23,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185036 s, 55.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:24,405::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:24,406::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69727::DEBUG::2014-09-25 17:10:24,408::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:24,847::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69728::DEBUG::2014-09-25 17:10:24,848::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:10:24,848::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69728::DEBUG::2014-09-25 17:10:24,850::task::595::Storage.TaskManager.Task::(_updateState) Task=`3f56d072-33b7-4f13-93aa-25f3b8f23843`::moving from state init -> state preparing >Thread-69728::INFO::2014-09-25 17:10:24,851::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69728::INFO::2014-09-25 17:10:24,851::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69728::DEBUG::2014-09-25 17:10:24,852::task::1191::Storage.TaskManager.Task::(prepare) Task=`3f56d072-33b7-4f13-93aa-25f3b8f23843`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69728::DEBUG::2014-09-25 17:10:24,852::task::595::Storage.TaskManager.Task::(_updateState) Task=`3f56d072-33b7-4f13-93aa-25f3b8f23843`::moving from state preparing -> state finished >Thread-69728::DEBUG::2014-09-25 17:10:24,852::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69728::DEBUG::2014-09-25 17:10:24,852::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69728::DEBUG::2014-09-25 17:10:24,852::task::993::Storage.TaskManager.Task::(_decref) Task=`3f56d072-33b7-4f13-93aa-25f3b8f23843`::ref 0 aborting False >Thread-69728::DEBUG::2014-09-25 17:10:24,853::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69728::DEBUG::2014-09-25 17:10:24,853::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:24,867::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69729::DEBUG::2014-09-25 17:10:24,868::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:10:24,868::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69729::DEBUG::2014-09-25 17:10:24,870::task::595::Storage.TaskManager.Task::(_updateState) Task=`f90472a7-c598-4182-ae3b-ffc25386dc13`::moving from state init -> state preparing >Thread-69729::INFO::2014-09-25 17:10:24,870::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69729::DEBUG::2014-09-25 17:10:24,871::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8541ea93-efe3-42bb-a584-c9619a626c14`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69729::DEBUG::2014-09-25 17:10:24,871::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69729::DEBUG::2014-09-25 17:10:24,872::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69729::DEBUG::2014-09-25 17:10:24,872::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8541ea93-efe3-42bb-a584-c9619a626c14`::Granted request >Thread-69729::DEBUG::2014-09-25 17:10:24,872::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f90472a7-c598-4182-ae3b-ffc25386dc13`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69729::DEBUG::2014-09-25 17:10:24,873::task::993::Storage.TaskManager.Task::(_decref) Task=`f90472a7-c598-4182-ae3b-ffc25386dc13`::ref 1 aborting False >Thread-69729::INFO::2014-09-25 17:10:24,877::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69729::DEBUG::2014-09-25 17:10:24,877::task::1191::Storage.TaskManager.Task::(prepare) Task=`f90472a7-c598-4182-ae3b-ffc25386dc13`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69729::DEBUG::2014-09-25 17:10:24,877::task::595::Storage.TaskManager.Task::(_updateState) Task=`f90472a7-c598-4182-ae3b-ffc25386dc13`::moving from state preparing -> state finished >Thread-69729::DEBUG::2014-09-25 17:10:24,878::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69729::DEBUG::2014-09-25 17:10:24,878::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69729::DEBUG::2014-09-25 17:10:24,878::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69729::DEBUG::2014-09-25 17:10:24,879::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69729::DEBUG::2014-09-25 17:10:24,879::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69729::DEBUG::2014-09-25 17:10:24,879::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69729::DEBUG::2014-09-25 17:10:24,879::task::993::Storage.TaskManager.Task::(_decref) Task=`f90472a7-c598-4182-ae3b-ffc25386dc13`::ref 0 aborting False >Thread-69729::DEBUG::2014-09-25 17:10:24,880::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69729::DEBUG::2014-09-25 17:10:24,880::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:25,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:25,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202593 s, 50.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:26,137::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:26,137::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:26,137::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:26,137::__init__::232::IOProcess::(_processLogs) (107248) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:26,144::__init__::232::IOProcess::(_processLogs) (107248) Queuing response >Thread-81::DEBUG::2014-09-25 17:10:26,145::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:10:26,162::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00765316 s, 48.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:26,163::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:26,164::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:26,164::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:26,164::__init__::232::IOProcess::(_processLogs) (107249) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:26,168::__init__::232::IOProcess::(_processLogs) (107249) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:27,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:27,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0226389 s, 45.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:27,423::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:27,426::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69730::DEBUG::2014-09-25 17:10:27,431::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:29,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:29,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155163 s, 66.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:30,446::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:30,448::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69731::DEBUG::2014-09-25 17:10:30,450::task::595::Storage.TaskManager.Task::(_updateState) Task=`3175fbf7-ec82-4ec6-8ce5-1f642a2e0e09`::moving from state init -> state preparing >Thread-69731::INFO::2014-09-25 17:10:30,451::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69731::INFO::2014-09-25 17:10:30,451::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00875', 'lastCheck': '9.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000519427', 'lastCheck': '8.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00765316', 'lastCheck': '4.3', 'valid': True}} >Thread-69731::DEBUG::2014-09-25 17:10:30,451::task::1191::Storage.TaskManager.Task::(prepare) Task=`3175fbf7-ec82-4ec6-8ce5-1f642a2e0e09`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00875', 'lastCheck': '9.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000519427', 'lastCheck': '8.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00765316', 'lastCheck': '4.3', 'valid': True}} >Thread-69731::DEBUG::2014-09-25 17:10:30,452::task::595::Storage.TaskManager.Task::(_updateState) Task=`3175fbf7-ec82-4ec6-8ce5-1f642a2e0e09`::moving from state preparing -> state finished >Thread-69731::DEBUG::2014-09-25 17:10:30,452::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69731::DEBUG::2014-09-25 17:10:30,452::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69731::DEBUG::2014-09-25 17:10:30,452::task::993::Storage.TaskManager.Task::(_decref) Task=`3175fbf7-ec82-4ec6-8ce5-1f642a2e0e09`::ref 0 aborting False >Thread-69731::DEBUG::2014-09-25 17:10:30,455::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:30,476::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:30,477::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69732::DEBUG::2014-09-25 17:10:30,479::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:10:30,579::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:30,580::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:30,580::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:30,580::__init__::232::IOProcess::(_processLogs) (107250) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:30,580::__init__::232::IOProcess::(_processLogs) (107250) Queuing response >Thread-79::DEBUG::2014-09-25 17:10:30,582::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:10:30,591::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00036871 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:30,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:30,592::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:30,593::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:30,594::__init__::232::IOProcess::(_processLogs) (107251) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:30,594::__init__::232::IOProcess::(_processLogs) (107251) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:30,595::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:30,595::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:30,596::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:30,596::__init__::232::IOProcess::(_processLogs) (107252) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:30,596::__init__::232::IOProcess::(_processLogs) (107252) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:30,597::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:30,597::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:30,598::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:30,598::__init__::232::IOProcess::(_processLogs) (107253) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:30,598::__init__::232::IOProcess::(_processLogs) (107253) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:30,599::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:30,599::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:30,600::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:30,600::__init__::232::IOProcess::(_processLogs) (107254) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:30,601::__init__::232::IOProcess::(_processLogs) (107254) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:31,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:31,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0313554 s, 32.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:32,184::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:32,185::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:32,185::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:32,185::__init__::232::IOProcess::(_processLogs) (107255) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:32,186::__init__::232::IOProcess::(_processLogs) (107255) Queuing response >Thread-80::DEBUG::2014-09-25 17:10:32,187::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:10:32,195::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000443131 s, 853 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:32,197::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:32,197::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:32,197::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:32,197::__init__::232::IOProcess::(_processLogs) (107256) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:32,198::__init__::232::IOProcess::(_processLogs) (107256) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:33,541::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:33,543::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69733::DEBUG::2014-09-25 17:10:33,545::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:33,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:33,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.271421 s, 3.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:34,884::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69734::DEBUG::2014-09-25 17:10:34,886::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StorageDomain.getFileStats' in bridge with {u'caseSensitive': False, u'pattern': u'*.vfd', u'storagedomainID': u'fc587b6a-bac3-4668-99de-7f6fd6acce7f'} >JsonRpcServer::DEBUG::2014-09-25 17:10:34,886::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69734::DEBUG::2014-09-25 17:10:34,888::task::595::Storage.TaskManager.Task::(_updateState) Task=`708a1082-3027-4850-900d-cf1a8b90d009`::moving from state init -> state preparing >Thread-69734::INFO::2014-09-25 17:10:34,888::logUtils::44::dispatcher::(wrapper) Run and protect: getFileStats(sdUUID=u'fc587b6a-bac3-4668-99de-7f6fd6acce7f', pattern=u'*.vfd', caseSensitive=False, options=None) >Thread-69734::DEBUG::2014-09-25 17:10:34,889::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f`ReqID=`00536e9b-cbef-4d8c-be39-603e5a9ce0c3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2301' at 'getFileStats' >Thread-69734::DEBUG::2014-09-25 17:10:34,889::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f' for lock type 'shared' >Thread-69734::DEBUG::2014-09-25 17:10:34,890::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f' is free. Now locking as 'shared' (1 active user) >Thread-69734::DEBUG::2014-09-25 17:10:34,890::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f`ReqID=`00536e9b-cbef-4d8c-be39-603e5a9ce0c3`::Granted request >Thread-69734::DEBUG::2014-09-25 17:10:34,891::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`708a1082-3027-4850-900d-cf1a8b90d009`::_resourcesAcquired: Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f (shared) >Thread-69734::DEBUG::2014-09-25 17:10:34,891::task::993::Storage.TaskManager.Task::(_decref) Task=`708a1082-3027-4850-900d-cf1a8b90d009`::ref 1 aborting False >Thread-29::DEBUG::2014-09-25 17:10:34,892::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,892::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,892::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,892::__init__::232::IOProcess::(_processLogs) (107257) Got request for method 'listdir' >Thread-29::DEBUG::2014-09-25 17:10:34,893::__init__::232::IOProcess::(_processLogs) (107257) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,894::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,894::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,894::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,894::__init__::232::IOProcess::(_processLogs) (107258) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,895::__init__::232::IOProcess::(_processLogs) (107258) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,896::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,896::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,897::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,897::__init__::232::IOProcess::(_processLogs) (107259) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,897::__init__::232::IOProcess::(_processLogs) (107259) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,898::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,898::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,899::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,899::__init__::232::IOProcess::(_processLogs) (107260) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,900::__init__::232::IOProcess::(_processLogs) (107260) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,900::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,900::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,901::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,901::__init__::232::IOProcess::(_processLogs) (107261) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,902::__init__::232::IOProcess::(_processLogs) (107261) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,902::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,903::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,903::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,904::__init__::232::IOProcess::(_processLogs) (107262) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,904::__init__::232::IOProcess::(_processLogs) (107262) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,905::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,905::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,906::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,906::__init__::232::IOProcess::(_processLogs) (107263) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,907::__init__::232::IOProcess::(_processLogs) (107263) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,907::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,907::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,908::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,908::__init__::232::IOProcess::(_processLogs) (107264) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,909::__init__::232::IOProcess::(_processLogs) (107264) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,909::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,909::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,910::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,911::__init__::232::IOProcess::(_processLogs) (107265) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,911::__init__::232::IOProcess::(_processLogs) (107265) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,911::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,912::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,912::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-69734::INFO::2014-09-25 17:10:34,912::logUtils::47::dispatcher::(wrapper) Run and protect: getFileStats, Return response: {'fileStats': {u'virtio-win_x86.vfd': {'status': 0, 'ctime': '1410269074.0', 'size': '2949120'}, u'virtio-win_amd64.vfd': {'status': 0, 'ctime': '1410269074.0', 'size': '2949120'}}} >Thread-29::DEBUG::2014-09-25 17:10:34,913::__init__::232::IOProcess::(_processLogs) (107266) Got request for method 'stat' >Thread-69734::DEBUG::2014-09-25 17:10:34,914::task::1191::Storage.TaskManager.Task::(prepare) Task=`708a1082-3027-4850-900d-cf1a8b90d009`::finished: {'fileStats': {u'virtio-win_x86.vfd': {'status': 0, 'ctime': '1410269074.0', 'size': '2949120'}, u'virtio-win_amd64.vfd': {'status': 0, 'ctime': '1410269074.0', 'size': '2949120'}}} >Thread-29::DEBUG::2014-09-25 17:10:34,915::__init__::232::IOProcess::(_processLogs) (107266) Queuing response >Thread-69734::DEBUG::2014-09-25 17:10:34,915::task::595::Storage.TaskManager.Task::(_updateState) Task=`708a1082-3027-4850-900d-cf1a8b90d009`::moving from state preparing -> state finished >Thread-69734::DEBUG::2014-09-25 17:10:34,916::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f': < ResourceRef 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f', isValid: 'True' obj: 'None'>} >Thread-69734::DEBUG::2014-09-25 17:10:34,916::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69734::DEBUG::2014-09-25 17:10:34,916::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f' >Thread-69734::DEBUG::2014-09-25 17:10:34,917::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f' (0 active users) >Thread-69734::DEBUG::2014-09-25 17:10:34,917::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f' is free, finding out if anyone is waiting for it. >Thread-69734::DEBUG::2014-09-25 17:10:34,917::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f', Clearing records. >Thread-69734::DEBUG::2014-09-25 17:10:34,918::task::993::Storage.TaskManager.Task::(_decref) Task=`708a1082-3027-4850-900d-cf1a8b90d009`::ref 0 aborting False >Thread-69734::DEBUG::2014-09-25 17:10:34,918::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StorageDomain.getFileStats' in bridge with {u'virtio-win_x86.vfd': {'status': 0, 'ctime': '1410269074.0', 'size': '2949120'}, u'virtio-win_amd64.vfd': {'status': 0, 'ctime': '1410269074.0', 'size': '2949120'}} >Thread-69734::DEBUG::2014-09-25 17:10:34,918::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:34,922::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:34,924::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69735::DEBUG::2014-09-25 17:10:34,924::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69735::DEBUG::2014-09-25 17:10:34,925::task::595::Storage.TaskManager.Task::(_updateState) Task=`02262b17-c5cc-42d6-94c9-5c9f2a1d9dc2`::moving from state init -> state preparing >Thread-69735::INFO::2014-09-25 17:10:34,926::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69735::INFO::2014-09-25 17:10:34,926::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69735::DEBUG::2014-09-25 17:10:34,926::task::1191::Storage.TaskManager.Task::(prepare) Task=`02262b17-c5cc-42d6-94c9-5c9f2a1d9dc2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69735::DEBUG::2014-09-25 17:10:34,927::task::595::Storage.TaskManager.Task::(_updateState) Task=`02262b17-c5cc-42d6-94c9-5c9f2a1d9dc2`::moving from state preparing -> state finished >Thread-69735::DEBUG::2014-09-25 17:10:34,927::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69735::DEBUG::2014-09-25 17:10:34,927::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69735::DEBUG::2014-09-25 17:10:34,927::task::993::Storage.TaskManager.Task::(_decref) Task=`02262b17-c5cc-42d6-94c9-5c9f2a1d9dc2`::ref 0 aborting False >Thread-69735::DEBUG::2014-09-25 17:10:34,928::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69735::DEBUG::2014-09-25 17:10:34,928::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:34,938::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:34,939::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69736::DEBUG::2014-09-25 17:10:34,939::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69736::DEBUG::2014-09-25 17:10:34,941::task::595::Storage.TaskManager.Task::(_updateState) Task=`f0553cdb-c010-4f88-b031-d9500b6ccbfe`::moving from state init -> state preparing >Thread-69736::INFO::2014-09-25 17:10:34,941::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69736::DEBUG::2014-09-25 17:10:34,942::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ec9f56df-a63b-4af7-8313-1fa479b1b1b7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69736::DEBUG::2014-09-25 17:10:34,942::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69736::DEBUG::2014-09-25 17:10:34,942::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69736::DEBUG::2014-09-25 17:10:34,943::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ec9f56df-a63b-4af7-8313-1fa479b1b1b7`::Granted request >Thread-69736::DEBUG::2014-09-25 17:10:34,943::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f0553cdb-c010-4f88-b031-d9500b6ccbfe`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69736::DEBUG::2014-09-25 17:10:34,944::task::993::Storage.TaskManager.Task::(_decref) Task=`f0553cdb-c010-4f88-b031-d9500b6ccbfe`::ref 1 aborting False >Thread-69736::INFO::2014-09-25 17:10:34,948::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69736::DEBUG::2014-09-25 17:10:34,948::task::1191::Storage.TaskManager.Task::(prepare) Task=`f0553cdb-c010-4f88-b031-d9500b6ccbfe`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69736::DEBUG::2014-09-25 17:10:34,948::task::595::Storage.TaskManager.Task::(_updateState) Task=`f0553cdb-c010-4f88-b031-d9500b6ccbfe`::moving from state preparing -> state finished >Thread-69736::DEBUG::2014-09-25 17:10:34,949::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69736::DEBUG::2014-09-25 17:10:34,949::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69736::DEBUG::2014-09-25 17:10:34,949::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69736::DEBUG::2014-09-25 17:10:34,950::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69736::DEBUG::2014-09-25 17:10:34,950::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69736::DEBUG::2014-09-25 17:10:34,950::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69736::DEBUG::2014-09-25 17:10:34,950::task::993::Storage.TaskManager.Task::(_decref) Task=`f0553cdb-c010-4f88-b031-d9500b6ccbfe`::ref 0 aborting False >Thread-69736::DEBUG::2014-09-25 17:10:34,951::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69736::DEBUG::2014-09-25 17:10:34,951::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:34,970::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:34,971::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69737::DEBUG::2014-09-25 17:10:34,971::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StorageDomain.getFileStats' in bridge with {u'caseSensitive': False, u'pattern': u'*.iso', u'storagedomainID': u'fc587b6a-bac3-4668-99de-7f6fd6acce7f'} >Thread-69737::DEBUG::2014-09-25 17:10:34,973::task::595::Storage.TaskManager.Task::(_updateState) Task=`af1877d1-235c-42b2-ab2e-8d0e9c2c4e78`::moving from state init -> state preparing >Thread-69737::INFO::2014-09-25 17:10:34,973::logUtils::44::dispatcher::(wrapper) Run and protect: getFileStats(sdUUID=u'fc587b6a-bac3-4668-99de-7f6fd6acce7f', pattern=u'*.iso', caseSensitive=False, options=None) >Thread-69737::DEBUG::2014-09-25 17:10:34,974::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f`ReqID=`7ebdb1e3-a3e7-4585-8ced-64b16a27824c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2301' at 'getFileStats' >Thread-69737::DEBUG::2014-09-25 17:10:34,974::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f' for lock type 'shared' >Thread-69737::DEBUG::2014-09-25 17:10:34,974::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f' is free. Now locking as 'shared' (1 active user) >Thread-69737::DEBUG::2014-09-25 17:10:34,974::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f`ReqID=`7ebdb1e3-a3e7-4585-8ced-64b16a27824c`::Granted request >Thread-69737::DEBUG::2014-09-25 17:10:34,975::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`af1877d1-235c-42b2-ab2e-8d0e9c2c4e78`::_resourcesAcquired: Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f (shared) >Thread-69737::DEBUG::2014-09-25 17:10:34,975::task::993::Storage.TaskManager.Task::(_decref) Task=`af1877d1-235c-42b2-ab2e-8d0e9c2c4e78`::ref 1 aborting False >Thread-29::DEBUG::2014-09-25 17:10:34,976::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,976::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,977::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,977::__init__::232::IOProcess::(_processLogs) (107267) Got request for method 'listdir' >Thread-29::DEBUG::2014-09-25 17:10:34,977::__init__::232::IOProcess::(_processLogs) (107267) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,978::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,978::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,979::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,980::__init__::232::IOProcess::(_processLogs) (107268) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,980::__init__::232::IOProcess::(_processLogs) (107268) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,980::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,981::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,981::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,981::__init__::232::IOProcess::(_processLogs) (107269) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,981::__init__::232::IOProcess::(_processLogs) (107269) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,982::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,983::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,983::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,983::__init__::232::IOProcess::(_processLogs) (107270) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,983::__init__::232::IOProcess::(_processLogs) (107270) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,984::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,984::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,985::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,985::__init__::232::IOProcess::(_processLogs) (107271) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,985::__init__::232::IOProcess::(_processLogs) (107271) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,986::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,986::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,987::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,987::__init__::232::IOProcess::(_processLogs) (107272) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,987::__init__::232::IOProcess::(_processLogs) (107272) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,988::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,988::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,988::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,989::__init__::232::IOProcess::(_processLogs) (107273) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,989::__init__::232::IOProcess::(_processLogs) (107273) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,990::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,990::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,990::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,991::__init__::232::IOProcess::(_processLogs) (107274) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,991::__init__::232::IOProcess::(_processLogs) (107274) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,992::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,993::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:34,993::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:34,994::__init__::232::IOProcess::(_processLogs) (107275) Got request for method 'stat' >Thread-29::DEBUG::2014-09-25 17:10:34,994::__init__::232::IOProcess::(_processLogs) (107275) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:34,994::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:34,995::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-69737::INFO::2014-09-25 17:10:34,995::logUtils::47::dispatcher::(wrapper) Run and protect: getFileStats, Return response: {'fileStats': {u'rhev-tools-setup.iso': {'status': 0, 'ctime': '1410269074.0', 'size': '350183424'}, u'virtio-win.iso': {'status': 0, 'ctime': '1410269074.0', 'size': '149004288'}}} >Thread-29::DEBUG::2014-09-25 17:10:34,996::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-69737::DEBUG::2014-09-25 17:10:34,996::task::1191::Storage.TaskManager.Task::(prepare) Task=`af1877d1-235c-42b2-ab2e-8d0e9c2c4e78`::finished: {'fileStats': {u'rhev-tools-setup.iso': {'status': 0, 'ctime': '1410269074.0', 'size': '350183424'}, u'virtio-win.iso': {'status': 0, 'ctime': '1410269074.0', 'size': '149004288'}}} >Thread-29::DEBUG::2014-09-25 17:10:34,997::__init__::232::IOProcess::(_processLogs) (107276) Got request for method 'stat' >Thread-69737::DEBUG::2014-09-25 17:10:34,998::task::595::Storage.TaskManager.Task::(_updateState) Task=`af1877d1-235c-42b2-ab2e-8d0e9c2c4e78`::moving from state preparing -> state finished >Thread-69737::DEBUG::2014-09-25 17:10:34,999::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f': < ResourceRef 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f', isValid: 'True' obj: 'None'>} >Thread-29::DEBUG::2014-09-25 17:10:34,999::__init__::232::IOProcess::(_processLogs) (107276) Queuing response >Thread-69737::DEBUG::2014-09-25 17:10:35,000::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69737::DEBUG::2014-09-25 17:10:35,000::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f' >Thread-69737::DEBUG::2014-09-25 17:10:35,001::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f' (0 active users) >Thread-69737::DEBUG::2014-09-25 17:10:35,001::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f' is free, finding out if anyone is waiting for it. >Thread-69737::DEBUG::2014-09-25 17:10:35,001::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.fc587b6a-bac3-4668-99de-7f6fd6acce7f', Clearing records. >Thread-69737::DEBUG::2014-09-25 17:10:35,001::task::993::Storage.TaskManager.Task::(_decref) Task=`af1877d1-235c-42b2-ab2e-8d0e9c2c4e78`::ref 0 aborting False >Thread-69737::DEBUG::2014-09-25 17:10:35,002::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StorageDomain.getFileStats' in bridge with {u'rhev-tools-setup.iso': {'status': 0, 'ctime': '1410269074.0', 'size': '350183424'}, u'virtio-win.iso': {'status': 0, 'ctime': '1410269074.0', 'size': '149004288'}} >Thread-69737::DEBUG::2014-09-25 17:10:35,002::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:35,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:35,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133992 s, 76.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:36,171::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:36,171::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:36,172::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:10:36,173::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:10:36,174::__init__::232::IOProcess::(_processLogs) (107277) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:36,182::__init__::232::IOProcess::(_processLogs) (107277) Queuing response >Thread-81::DEBUG::2014-09-25 17:10:36,183::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000615528 s, 603 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:36,184::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:36,184::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:36,184::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:36,185::__init__::232::IOProcess::(_processLogs) (107278) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:36,185::__init__::232::IOProcess::(_processLogs) (107278) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:36,560::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:36,561::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69738::DEBUG::2014-09-25 17:10:36,563::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:37,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:37,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162782 s, 62.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:39,576::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:39,577::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69739::DEBUG::2014-09-25 17:10:39,579::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:39,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:39,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201086 s, 50.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:40,601::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:40,601::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:40,602::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:40,602::__init__::232::IOProcess::(_processLogs) (107279) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:40,605::__init__::232::IOProcess::(_processLogs) (107279) Queuing response >Thread-79::DEBUG::2014-09-25 17:10:40,606::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:10:40,624::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00847478 s, 66.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:40,625::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:40,625::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:40,625::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:40,626::__init__::232::IOProcess::(_processLogs) (107280) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:40,627::__init__::232::IOProcess::(_processLogs) (107280) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:40,629::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:40,629::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:40,629::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:40,629::__init__::232::IOProcess::(_processLogs) (107281) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:40,636::__init__::232::IOProcess::(_processLogs) (107281) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:40,637::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:40,637::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:40,637::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:40,638::__init__::232::IOProcess::(_processLogs) (107282) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:40,643::__init__::232::IOProcess::(_processLogs) (107282) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:40,644::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:40,644::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:40,644::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:40,644::__init__::232::IOProcess::(_processLogs) (107283) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:40,647::__init__::232::IOProcess::(_processLogs) (107283) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:42,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:42,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181206 s, 56.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:42,201::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:42,201::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:42,201::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:42,201::__init__::232::IOProcess::(_processLogs) (107284) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:42,202::__init__::232::IOProcess::(_processLogs) (107284) Queuing response >Thread-80::DEBUG::2014-09-25 17:10:42,203::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:10:42,212::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000357287 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:42,213::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:42,213::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:42,214::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:42,215::__init__::232::IOProcess::(_processLogs) (107285) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:42,215::__init__::232::IOProcess::(_processLogs) (107285) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:42,591::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:42,593::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69740::DEBUG::2014-09-25 17:10:42,595::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:44,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:44,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172101 s, 59.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:44,975::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69741::DEBUG::2014-09-25 17:10:44,976::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:10:44,976::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69741::DEBUG::2014-09-25 17:10:44,979::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f398c6f-6051-47c0-b822-d67a200088c7`::moving from state init -> state preparing >Thread-69741::INFO::2014-09-25 17:10:44,979::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69741::INFO::2014-09-25 17:10:44,980::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69741::DEBUG::2014-09-25 17:10:44,980::task::1191::Storage.TaskManager.Task::(prepare) Task=`4f398c6f-6051-47c0-b822-d67a200088c7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69741::DEBUG::2014-09-25 17:10:44,980::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f398c6f-6051-47c0-b822-d67a200088c7`::moving from state preparing -> state finished >Thread-69741::DEBUG::2014-09-25 17:10:44,980::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69741::DEBUG::2014-09-25 17:10:44,980::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69741::DEBUG::2014-09-25 17:10:44,981::task::993::Storage.TaskManager.Task::(_decref) Task=`4f398c6f-6051-47c0-b822-d67a200088c7`::ref 0 aborting False >Thread-69741::DEBUG::2014-09-25 17:10:44,981::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69741::DEBUG::2014-09-25 17:10:44,981::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:44,992::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:44,993::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69742::DEBUG::2014-09-25 17:10:44,993::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69742::DEBUG::2014-09-25 17:10:44,995::task::595::Storage.TaskManager.Task::(_updateState) Task=`de55b68f-6ee0-4410-8ce4-d6cd06035323`::moving from state init -> state preparing >Thread-69742::INFO::2014-09-25 17:10:44,995::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69742::DEBUG::2014-09-25 17:10:44,996::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4a407d66-4cfc-49ad-a3ea-a7a3885f440c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69742::DEBUG::2014-09-25 17:10:44,996::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69742::DEBUG::2014-09-25 17:10:44,996::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69742::DEBUG::2014-09-25 17:10:44,997::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4a407d66-4cfc-49ad-a3ea-a7a3885f440c`::Granted request >Thread-69742::DEBUG::2014-09-25 17:10:44,997::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`de55b68f-6ee0-4410-8ce4-d6cd06035323`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69742::DEBUG::2014-09-25 17:10:44,998::task::993::Storage.TaskManager.Task::(_decref) Task=`de55b68f-6ee0-4410-8ce4-d6cd06035323`::ref 1 aborting False >Thread-69742::INFO::2014-09-25 17:10:45,002::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69742::DEBUG::2014-09-25 17:10:45,002::task::1191::Storage.TaskManager.Task::(prepare) Task=`de55b68f-6ee0-4410-8ce4-d6cd06035323`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69742::DEBUG::2014-09-25 17:10:45,003::task::595::Storage.TaskManager.Task::(_updateState) Task=`de55b68f-6ee0-4410-8ce4-d6cd06035323`::moving from state preparing -> state finished >Thread-69742::DEBUG::2014-09-25 17:10:45,003::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69742::DEBUG::2014-09-25 17:10:45,003::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69742::DEBUG::2014-09-25 17:10:45,003::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69742::DEBUG::2014-09-25 17:10:45,004::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69742::DEBUG::2014-09-25 17:10:45,004::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69742::DEBUG::2014-09-25 17:10:45,004::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69742::DEBUG::2014-09-25 17:10:45,005::task::993::Storage.TaskManager.Task::(_decref) Task=`de55b68f-6ee0-4410-8ce4-d6cd06035323`::ref 0 aborting False >Thread-69742::DEBUG::2014-09-25 17:10:45,005::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69742::DEBUG::2014-09-25 17:10:45,006::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:45,606::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:45,607::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69743::DEBUG::2014-09-25 17:10:45,610::task::595::Storage.TaskManager.Task::(_updateState) Task=`252cba80-d964-4760-a3c9-79b490ea509a`::moving from state init -> state preparing >Thread-69743::INFO::2014-09-25 17:10:45,610::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69743::INFO::2014-09-25 17:10:45,611::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00847478', 'lastCheck': '5.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000357287', 'lastCheck': '3.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000615528', 'lastCheck': '9.4', 'valid': True}} >Thread-69743::DEBUG::2014-09-25 17:10:45,611::task::1191::Storage.TaskManager.Task::(prepare) Task=`252cba80-d964-4760-a3c9-79b490ea509a`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00847478', 'lastCheck': '5.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000357287', 'lastCheck': '3.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000615528', 'lastCheck': '9.4', 'valid': True}} >Thread-69743::DEBUG::2014-09-25 17:10:45,611::task::595::Storage.TaskManager.Task::(_updateState) Task=`252cba80-d964-4760-a3c9-79b490ea509a`::moving from state preparing -> state finished >Thread-69743::DEBUG::2014-09-25 17:10:45,611::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69743::DEBUG::2014-09-25 17:10:45,612::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69743::DEBUG::2014-09-25 17:10:45,612::task::993::Storage.TaskManager.Task::(_decref) Task=`252cba80-d964-4760-a3c9-79b490ea509a`::ref 0 aborting False >Thread-69743::DEBUG::2014-09-25 17:10:45,614::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:45,631::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:45,632::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69744::DEBUG::2014-09-25 17:10:45,634::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:46,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:46,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0304068 s, 33.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:46,189::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:46,191::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:46,193::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:46,194::__init__::232::IOProcess::(_processLogs) (107286) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:46,197::__init__::232::IOProcess::(_processLogs) (107286) Queuing response >Thread-81::DEBUG::2014-09-25 17:10:46,199::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:10:46,231::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0127617 s, 29.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:46,235::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:46,237::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:46,239::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:46,242::__init__::232::IOProcess::(_processLogs) (107287) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:46,244::__init__::232::IOProcess::(_processLogs) (107287) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:48,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:48,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.022867 s, 44.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:48,675::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:48,676::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69745::DEBUG::2014-09-25 17:10:48,678::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:50,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:50,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158218 s, 64.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:50,649::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:50,649::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:50,649::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:50,650::__init__::232::IOProcess::(_processLogs) (107288) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:50,657::__init__::232::IOProcess::(_processLogs) (107288) Queuing response >Thread-79::DEBUG::2014-09-25 17:10:50,659::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:10:50,679::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00606446 s, 92.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:50,680::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:50,680::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:50,681::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:50,681::__init__::232::IOProcess::(_processLogs) (107289) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:50,689::__init__::232::IOProcess::(_processLogs) (107289) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:50,690::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:50,690::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:50,691::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:50,692::__init__::232::IOProcess::(_processLogs) (107290) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:50,692::__init__::232::IOProcess::(_processLogs) (107290) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:50,692::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:50,693::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:50,693::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:50,693::__init__::232::IOProcess::(_processLogs) (107291) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:50,694::__init__::232::IOProcess::(_processLogs) (107291) Queuing response >Thread-29::DEBUG::2014-09-25 17:10:50,695::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:50,695::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:50,695::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:50,695::__init__::232::IOProcess::(_processLogs) (107292) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:10:50,696::__init__::232::IOProcess::(_processLogs) (107292) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:51,694::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:51,695::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69746::DEBUG::2014-09-25 17:10:51,698::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:10:52,216::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:52,216::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:52,216::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:52,217::__init__::232::IOProcess::(_processLogs) (107293) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:52,217::__init__::232::IOProcess::(_processLogs) (107293) Queuing response >Thread-80::DEBUG::2014-09-25 17:10:52,218::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:10:52,227::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00047867 s, 790 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:52,228::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:52,228::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:52,229::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:52,229::__init__::232::IOProcess::(_processLogs) (107294) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:52,230::__init__::232::IOProcess::(_processLogs) (107294) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:52,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:52,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213532 s, 48.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:10:54,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:54,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165148 s, 62.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:54,717::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:54,719::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69747::DEBUG::2014-09-25 17:10:54,721::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:55,033::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69748::DEBUG::2014-09-25 17:10:55,034::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:10:55,034::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69748::DEBUG::2014-09-25 17:10:55,036::task::595::Storage.TaskManager.Task::(_updateState) Task=`9ccdaa5f-8ab3-499d-a9c2-0d2603a63615`::moving from state init -> state preparing >Thread-69748::INFO::2014-09-25 17:10:55,037::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69748::INFO::2014-09-25 17:10:55,037::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69748::DEBUG::2014-09-25 17:10:55,038::task::1191::Storage.TaskManager.Task::(prepare) Task=`9ccdaa5f-8ab3-499d-a9c2-0d2603a63615`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69748::DEBUG::2014-09-25 17:10:55,038::task::595::Storage.TaskManager.Task::(_updateState) Task=`9ccdaa5f-8ab3-499d-a9c2-0d2603a63615`::moving from state preparing -> state finished >Thread-69748::DEBUG::2014-09-25 17:10:55,038::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69748::DEBUG::2014-09-25 17:10:55,038::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69748::DEBUG::2014-09-25 17:10:55,038::task::993::Storage.TaskManager.Task::(_decref) Task=`9ccdaa5f-8ab3-499d-a9c2-0d2603a63615`::ref 0 aborting False >Thread-69748::DEBUG::2014-09-25 17:10:55,039::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69748::DEBUG::2014-09-25 17:10:55,039::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:55,053::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69749::DEBUG::2014-09-25 17:10:55,054::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:10:55,054::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69749::DEBUG::2014-09-25 17:10:55,056::task::595::Storage.TaskManager.Task::(_updateState) Task=`428a32a1-0711-4962-980a-23e58665b7d4`::moving from state init -> state preparing >Thread-69749::INFO::2014-09-25 17:10:55,057::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69749::DEBUG::2014-09-25 17:10:55,057::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`dbfc08e8-ac68-4494-a29f-163eaf159084`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69749::DEBUG::2014-09-25 17:10:55,058::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69749::DEBUG::2014-09-25 17:10:55,058::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69749::DEBUG::2014-09-25 17:10:55,058::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`dbfc08e8-ac68-4494-a29f-163eaf159084`::Granted request >Thread-69749::DEBUG::2014-09-25 17:10:55,059::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`428a32a1-0711-4962-980a-23e58665b7d4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69749::DEBUG::2014-09-25 17:10:55,059::task::993::Storage.TaskManager.Task::(_decref) Task=`428a32a1-0711-4962-980a-23e58665b7d4`::ref 1 aborting False >Thread-69749::INFO::2014-09-25 17:10:55,063::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69749::DEBUG::2014-09-25 17:10:55,064::task::1191::Storage.TaskManager.Task::(prepare) Task=`428a32a1-0711-4962-980a-23e58665b7d4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69749::DEBUG::2014-09-25 17:10:55,064::task::595::Storage.TaskManager.Task::(_updateState) Task=`428a32a1-0711-4962-980a-23e58665b7d4`::moving from state preparing -> state finished >Thread-69749::DEBUG::2014-09-25 17:10:55,064::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69749::DEBUG::2014-09-25 17:10:55,064::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69749::DEBUG::2014-09-25 17:10:55,065::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69749::DEBUG::2014-09-25 17:10:55,065::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69749::DEBUG::2014-09-25 17:10:55,065::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69749::DEBUG::2014-09-25 17:10:55,065::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69749::DEBUG::2014-09-25 17:10:55,066::task::993::Storage.TaskManager.Task::(_decref) Task=`428a32a1-0711-4962-980a-23e58665b7d4`::ref 0 aborting False >Thread-69749::DEBUG::2014-09-25 17:10:55,066::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69749::DEBUG::2014-09-25 17:10:55,067::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:10:56,243::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:56,243::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:56,243::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:56,243::__init__::232::IOProcess::(_processLogs) (107295) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:56,263::__init__::232::IOProcess::(_processLogs) (107295) Queuing response >Thread-81::DEBUG::2014-09-25 17:10:56,265::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:10:56,315::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0127295 s, 29.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:10:56,316::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:10:56,317::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:10:56,317::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:10:56,317::__init__::232::IOProcess::(_processLogs) (107296) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:10:56,323::__init__::232::IOProcess::(_processLogs) (107296) Queuing response >Dummy-148::DEBUG::2014-09-25 17:10:56,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:56,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0448468 s, 22.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:10:57,739::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:10:57,741::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69750::DEBUG::2014-09-25 17:10:57,743::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:10:58,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:10:58,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182836 s, 56.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:00,697::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:00,698::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:00,698::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:00,698::__init__::232::IOProcess::(_processLogs) (107297) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:00,700::__init__::232::IOProcess::(_processLogs) (107297) Queuing response >Thread-79::DEBUG::2014-09-25 17:11:00,701::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:00,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-79::DEBUG::2014-09-25 17:11:00,721::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00252369 s, 223 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:00,722::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:00,722::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:00,723::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:00,723::__init__::232::IOProcess::(_processLogs) (107298) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:00,738::__init__::232::IOProcess::(_processLogs) (107298) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:00,741::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:00,742::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:00,743::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:00,744::__init__::232::IOProcess::(_processLogs) (107299) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:00,745::__init__::232::IOProcess::(_processLogs) (107299) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:00,748::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:00,749::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:00,751::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:00,752::__init__::232::IOProcess::(_processLogs) (107300) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:00,753::__init__::232::IOProcess::(_processLogs) (107300) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:00,755::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:00,756::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:00,758::__init__::232::IOProcess::(_processLogs) Extracting request information... >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:00,760::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Dummy-148::DEBUG::2014-09-25 17:11:00,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0345029 s, 29.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:00,761::__init__::232::IOProcess::(_processLogs) (107301) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:00,764::__init__::232::IOProcess::(_processLogs) (107301) Queuing response >JsonRpcServer::DEBUG::2014-09-25 17:11:00,765::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69751::DEBUG::2014-09-25 17:11:00,775::task::595::Storage.TaskManager.Task::(_updateState) Task=`d12dce84-3529-4826-a2e7-6268eed1f711`::moving from state init -> state preparing >Thread-69751::INFO::2014-09-25 17:11:00,776::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69751::INFO::2014-09-25 17:11:00,777::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00252369', 'lastCheck': '0.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00047867', 'lastCheck': '8.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0127295', 'lastCheck': '4.5', 'valid': True}} >Thread-69751::DEBUG::2014-09-25 17:11:00,779::task::1191::Storage.TaskManager.Task::(prepare) Task=`d12dce84-3529-4826-a2e7-6268eed1f711`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00252369', 'lastCheck': '0.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00047867', 'lastCheck': '8.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0127295', 'lastCheck': '4.5', 'valid': True}} >Thread-69751::DEBUG::2014-09-25 17:11:00,779::task::595::Storage.TaskManager.Task::(_updateState) Task=`d12dce84-3529-4826-a2e7-6268eed1f711`::moving from state preparing -> state finished >Thread-69751::DEBUG::2014-09-25 17:11:00,780::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69751::DEBUG::2014-09-25 17:11:00,781::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69751::DEBUG::2014-09-25 17:11:00,782::task::993::Storage.TaskManager.Task::(_decref) Task=`d12dce84-3529-4826-a2e7-6268eed1f711`::ref 0 aborting False >Thread-69751::DEBUG::2014-09-25 17:11:00,790::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:00,813::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:00,815::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69752::DEBUG::2014-09-25 17:11:00,816::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:11:02,231::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:02,232::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:02,232::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:02,232::__init__::232::IOProcess::(_processLogs) (107302) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:02,233::__init__::232::IOProcess::(_processLogs) (107302) Queuing response >Thread-80::DEBUG::2014-09-25 17:11:02,234::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:11:02,243::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000518168 s, 729 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:02,244::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:02,245::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:02,245::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:02,245::__init__::232::IOProcess::(_processLogs) (107303) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:02,246::__init__::232::IOProcess::(_processLogs) (107303) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:02,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:02,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215131 s, 47.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:03,873::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:03,874::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69753::DEBUG::2014-09-25 17:11:03,877::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:04,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:04,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172993 s, 59.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:05,096::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69754::DEBUG::2014-09-25 17:11:05,098::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:11:05,098::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69754::DEBUG::2014-09-25 17:11:05,100::task::595::Storage.TaskManager.Task::(_updateState) Task=`32e229bb-4a0b-4216-9f89-975729d22bbf`::moving from state init -> state preparing >Thread-69754::INFO::2014-09-25 17:11:05,101::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69754::INFO::2014-09-25 17:11:05,101::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69754::DEBUG::2014-09-25 17:11:05,102::task::1191::Storage.TaskManager.Task::(prepare) Task=`32e229bb-4a0b-4216-9f89-975729d22bbf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69754::DEBUG::2014-09-25 17:11:05,102::task::595::Storage.TaskManager.Task::(_updateState) Task=`32e229bb-4a0b-4216-9f89-975729d22bbf`::moving from state preparing -> state finished >Thread-69754::DEBUG::2014-09-25 17:11:05,102::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69754::DEBUG::2014-09-25 17:11:05,102::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69754::DEBUG::2014-09-25 17:11:05,102::task::993::Storage.TaskManager.Task::(_decref) Task=`32e229bb-4a0b-4216-9f89-975729d22bbf`::ref 0 aborting False >Thread-69754::DEBUG::2014-09-25 17:11:05,103::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69754::DEBUG::2014-09-25 17:11:05,103::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:05,114::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69755::DEBUG::2014-09-25 17:11:05,115::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:11:05,115::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69755::DEBUG::2014-09-25 17:11:05,117::task::595::Storage.TaskManager.Task::(_updateState) Task=`aaca5836-5c30-488b-a44a-bed9a71defcb`::moving from state init -> state preparing >Thread-69755::INFO::2014-09-25 17:11:05,117::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69755::DEBUG::2014-09-25 17:11:05,118::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`730f19b4-cd5c-4b44-89af-aeedf92491d7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69755::DEBUG::2014-09-25 17:11:05,118::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69755::DEBUG::2014-09-25 17:11:05,119::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69755::DEBUG::2014-09-25 17:11:05,119::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`730f19b4-cd5c-4b44-89af-aeedf92491d7`::Granted request >Thread-69755::DEBUG::2014-09-25 17:11:05,119::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`aaca5836-5c30-488b-a44a-bed9a71defcb`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69755::DEBUG::2014-09-25 17:11:05,120::task::993::Storage.TaskManager.Task::(_decref) Task=`aaca5836-5c30-488b-a44a-bed9a71defcb`::ref 1 aborting False >Thread-69755::INFO::2014-09-25 17:11:05,124::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69755::DEBUG::2014-09-25 17:11:05,124::task::1191::Storage.TaskManager.Task::(prepare) Task=`aaca5836-5c30-488b-a44a-bed9a71defcb`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69755::DEBUG::2014-09-25 17:11:05,125::task::595::Storage.TaskManager.Task::(_updateState) Task=`aaca5836-5c30-488b-a44a-bed9a71defcb`::moving from state preparing -> state finished >Thread-69755::DEBUG::2014-09-25 17:11:05,125::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69755::DEBUG::2014-09-25 17:11:05,125::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69755::DEBUG::2014-09-25 17:11:05,125::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69755::DEBUG::2014-09-25 17:11:05,126::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69755::DEBUG::2014-09-25 17:11:05,126::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69755::DEBUG::2014-09-25 17:11:05,126::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69755::DEBUG::2014-09-25 17:11:05,127::task::993::Storage.TaskManager.Task::(_decref) Task=`aaca5836-5c30-488b-a44a-bed9a71defcb`::ref 0 aborting False >Thread-69755::DEBUG::2014-09-25 17:11:05,127::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69755::DEBUG::2014-09-25 17:11:05,127::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:11:06,325::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:06,325::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:06,325::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:06,325::__init__::232::IOProcess::(_processLogs) (107304) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:06,327::__init__::232::IOProcess::(_processLogs) (107304) Queuing response >Thread-81::DEBUG::2014-09-25 17:11:06,328::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:11:06,336::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00041596 s, 892 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:06,338::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:06,338::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:06,338::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:06,338::__init__::232::IOProcess::(_processLogs) (107305) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:06,339::__init__::232::IOProcess::(_processLogs) (107305) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:06,890::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:06,892::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69756::DEBUG::2014-09-25 17:11:06,894::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:06,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:06,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184889 s, 55.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:11:08,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:08,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183814 s, 55.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:09,906::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:09,907::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69757::DEBUG::2014-09-25 17:11:09,909::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:11:10,771::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:10,771::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:10,771::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:10,771::__init__::232::IOProcess::(_processLogs) (107306) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:10,785::__init__::232::IOProcess::(_processLogs) (107306) Queuing response >Thread-79::DEBUG::2014-09-25 17:11:10,786::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:11:10,812::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00249338 s, 225 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:10,814::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:10,814::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:10,814::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:10,814::__init__::232::IOProcess::(_processLogs) (107307) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:10,822::__init__::232::IOProcess::(_processLogs) (107307) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:10,823::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:10,824::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:10,825::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:10,825::__init__::232::IOProcess::(_processLogs) (107308) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:10,825::__init__::232::IOProcess::(_processLogs) (107308) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:10,826::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:10,826::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:10,827::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:10,827::__init__::232::IOProcess::(_processLogs) (107309) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:10,827::__init__::232::IOProcess::(_processLogs) (107309) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:10,828::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:10,828::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:10,829::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:10,829::__init__::232::IOProcess::(_processLogs) (107310) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:10,830::__init__::232::IOProcess::(_processLogs) (107310) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:11,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:11,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184943 s, 55.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:12,248::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:12,249::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:12,249::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:12,249::__init__::232::IOProcess::(_processLogs) (107311) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:12,250::__init__::232::IOProcess::(_processLogs) (107311) Queuing response >Thread-80::DEBUG::2014-09-25 17:11:12,251::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:11:12,260::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000552387 s, 684 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:12,261::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:12,261::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:12,262::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:12,262::__init__::232::IOProcess::(_processLogs) (107312) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:12,262::__init__::232::IOProcess::(_processLogs) (107312) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:12,922::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:12,924::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69758::DEBUG::2014-09-25 17:11:12,926::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:13,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:13,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194775 s, 52.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:11:15,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:15,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193367 s, 53.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:15,155::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69759::DEBUG::2014-09-25 17:11:15,159::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:11:15,159::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69759::DEBUG::2014-09-25 17:11:15,165::task::595::Storage.TaskManager.Task::(_updateState) Task=`e8dd32ab-f5c2-4ea1-8a92-1c210cec5003`::moving from state init -> state preparing >Thread-69759::INFO::2014-09-25 17:11:15,166::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69759::INFO::2014-09-25 17:11:15,168::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69759::DEBUG::2014-09-25 17:11:15,169::task::1191::Storage.TaskManager.Task::(prepare) Task=`e8dd32ab-f5c2-4ea1-8a92-1c210cec5003`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69759::DEBUG::2014-09-25 17:11:15,170::task::595::Storage.TaskManager.Task::(_updateState) Task=`e8dd32ab-f5c2-4ea1-8a92-1c210cec5003`::moving from state preparing -> state finished >Thread-69759::DEBUG::2014-09-25 17:11:15,171::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69759::DEBUG::2014-09-25 17:11:15,172::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69759::DEBUG::2014-09-25 17:11:15,173::task::993::Storage.TaskManager.Task::(_decref) Task=`e8dd32ab-f5c2-4ea1-8a92-1c210cec5003`::ref 0 aborting False >Thread-69759::DEBUG::2014-09-25 17:11:15,175::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69759::DEBUG::2014-09-25 17:11:15,176::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:15,196::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:15,197::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69760::DEBUG::2014-09-25 17:11:15,197::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69760::DEBUG::2014-09-25 17:11:15,199::task::595::Storage.TaskManager.Task::(_updateState) Task=`2957a6a6-b0a2-424c-be7f-10d437d2482d`::moving from state init -> state preparing >Thread-69760::INFO::2014-09-25 17:11:15,200::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69760::DEBUG::2014-09-25 17:11:15,200::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b4ead66f-b8bd-4114-a030-d0259bb07176`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69760::DEBUG::2014-09-25 17:11:15,201::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69760::DEBUG::2014-09-25 17:11:15,201::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69760::DEBUG::2014-09-25 17:11:15,201::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b4ead66f-b8bd-4114-a030-d0259bb07176`::Granted request >Thread-69760::DEBUG::2014-09-25 17:11:15,202::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2957a6a6-b0a2-424c-be7f-10d437d2482d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69760::DEBUG::2014-09-25 17:11:15,202::task::993::Storage.TaskManager.Task::(_decref) Task=`2957a6a6-b0a2-424c-be7f-10d437d2482d`::ref 1 aborting False >Thread-69760::INFO::2014-09-25 17:11:15,207::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69760::DEBUG::2014-09-25 17:11:15,207::task::1191::Storage.TaskManager.Task::(prepare) Task=`2957a6a6-b0a2-424c-be7f-10d437d2482d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69760::DEBUG::2014-09-25 17:11:15,208::task::595::Storage.TaskManager.Task::(_updateState) Task=`2957a6a6-b0a2-424c-be7f-10d437d2482d`::moving from state preparing -> state finished >Thread-69760::DEBUG::2014-09-25 17:11:15,208::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69760::DEBUG::2014-09-25 17:11:15,208::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69760::DEBUG::2014-09-25 17:11:15,208::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69760::DEBUG::2014-09-25 17:11:15,209::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69760::DEBUG::2014-09-25 17:11:15,209::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69760::DEBUG::2014-09-25 17:11:15,209::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69760::DEBUG::2014-09-25 17:11:15,210::task::993::Storage.TaskManager.Task::(_decref) Task=`2957a6a6-b0a2-424c-be7f-10d437d2482d`::ref 0 aborting False >Thread-69760::DEBUG::2014-09-25 17:11:15,210::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69760::DEBUG::2014-09-25 17:11:15,211::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:15,937::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:15,938::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69761::DEBUG::2014-09-25 17:11:15,941::task::595::Storage.TaskManager.Task::(_updateState) Task=`32ee3c12-8167-4a0d-86d7-38185990061e`::moving from state init -> state preparing >Thread-69761::INFO::2014-09-25 17:11:15,941::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69761::INFO::2014-09-25 17:11:15,942::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00249338', 'lastCheck': '5.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000552387', 'lastCheck': '3.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00041596', 'lastCheck': '9.6', 'valid': True}} >Thread-69761::DEBUG::2014-09-25 17:11:15,942::task::1191::Storage.TaskManager.Task::(prepare) Task=`32ee3c12-8167-4a0d-86d7-38185990061e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00249338', 'lastCheck': '5.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000552387', 'lastCheck': '3.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00041596', 'lastCheck': '9.6', 'valid': True}} >Thread-69761::DEBUG::2014-09-25 17:11:15,942::task::595::Storage.TaskManager.Task::(_updateState) Task=`32ee3c12-8167-4a0d-86d7-38185990061e`::moving from state preparing -> state finished >Thread-69761::DEBUG::2014-09-25 17:11:15,942::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69761::DEBUG::2014-09-25 17:11:15,943::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69761::DEBUG::2014-09-25 17:11:15,943::task::993::Storage.TaskManager.Task::(_decref) Task=`32ee3c12-8167-4a0d-86d7-38185990061e`::ref 0 aborting False >Thread-69761::DEBUG::2014-09-25 17:11:15,945::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:15,962::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:15,963::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69762::DEBUG::2014-09-25 17:11:15,965::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:11:16,342::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:16,343::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:16,345::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:11:16,347::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:11:16,348::__init__::232::IOProcess::(_processLogs) (107313) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:16,357::__init__::232::IOProcess::(_processLogs) (107313) Queuing response >Thread-81::DEBUG::2014-09-25 17:11:16,375::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00540571 s, 68.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:16,377::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:16,379::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:16,380::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:16,381::__init__::232::IOProcess::(_processLogs) (107314) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:16,383::__init__::232::IOProcess::(_processLogs) (107314) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:17,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:17,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0379265 s, 27.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:19,014::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:19,015::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69763::DEBUG::2014-09-25 17:11:19,017::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:19,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:19,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0208961 s, 49.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:20,830::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:20,830::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:20,831::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:20,831::__init__::232::IOProcess::(_processLogs) (107315) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:20,834::__init__::232::IOProcess::(_processLogs) (107315) Queuing response >Thread-79::DEBUG::2014-09-25 17:11:20,835::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:11:20,870::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.01729 s, 32.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:20,872::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:20,872::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:20,872::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:20,873::__init__::232::IOProcess::(_processLogs) (107316) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:20,879::__init__::232::IOProcess::(_processLogs) (107316) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:20,881::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:20,881::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:20,882::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:20,882::__init__::232::IOProcess::(_processLogs) (107317) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:20,882::__init__::232::IOProcess::(_processLogs) (107317) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:20,883::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:20,883::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:20,883::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:20,884::__init__::232::IOProcess::(_processLogs) (107318) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:20,884::__init__::232::IOProcess::(_processLogs) (107318) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:20,885::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:20,885::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:20,886::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:20,887::__init__::232::IOProcess::(_processLogs) (107319) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:20,887::__init__::232::IOProcess::(_processLogs) (107319) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:21,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:21,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015967 s, 64.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:22,030::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:22,032::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69764::DEBUG::2014-09-25 17:11:22,034::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:11:22,265::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:22,265::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:22,265::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:22,266::__init__::232::IOProcess::(_processLogs) (107320) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:22,266::__init__::232::IOProcess::(_processLogs) (107320) Queuing response >Thread-80::DEBUG::2014-09-25 17:11:22,267::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:11:22,276::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000432201 s, 875 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:22,277::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:22,278::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:22,278::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:22,278::__init__::232::IOProcess::(_processLogs) (107321) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:22,279::__init__::232::IOProcess::(_processLogs) (107321) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:23,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:23,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0220981 s, 46.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:25,049::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:25,050::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69765::DEBUG::2014-09-25 17:11:25,052::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:25,238::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:25,239::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69766::DEBUG::2014-09-25 17:11:25,239::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69766::DEBUG::2014-09-25 17:11:25,241::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c5e6a6e-ff14-4e8c-b4fd-69764503e530`::moving from state init -> state preparing >Thread-69766::INFO::2014-09-25 17:11:25,241::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69766::INFO::2014-09-25 17:11:25,242::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69766::DEBUG::2014-09-25 17:11:25,242::task::1191::Storage.TaskManager.Task::(prepare) Task=`2c5e6a6e-ff14-4e8c-b4fd-69764503e530`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69766::DEBUG::2014-09-25 17:11:25,242::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c5e6a6e-ff14-4e8c-b4fd-69764503e530`::moving from state preparing -> state finished >Thread-69766::DEBUG::2014-09-25 17:11:25,243::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69766::DEBUG::2014-09-25 17:11:25,243::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69766::DEBUG::2014-09-25 17:11:25,243::task::993::Storage.TaskManager.Task::(_decref) Task=`2c5e6a6e-ff14-4e8c-b4fd-69764503e530`::ref 0 aborting False >Thread-69766::DEBUG::2014-09-25 17:11:25,243::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69766::DEBUG::2014-09-25 17:11:25,244::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:25,253::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:25,254::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69767::DEBUG::2014-09-25 17:11:25,254::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69767::DEBUG::2014-09-25 17:11:25,256::task::595::Storage.TaskManager.Task::(_updateState) Task=`85857448-77b6-4bd1-920b-d17ac56b7c48`::moving from state init -> state preparing >Thread-69767::INFO::2014-09-25 17:11:25,256::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69767::DEBUG::2014-09-25 17:11:25,257::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`be24dcc6-1eb2-4836-8cb7-94f4761536c2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69767::DEBUG::2014-09-25 17:11:25,257::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69767::DEBUG::2014-09-25 17:11:25,258::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69767::DEBUG::2014-09-25 17:11:25,258::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`be24dcc6-1eb2-4836-8cb7-94f4761536c2`::Granted request >Thread-69767::DEBUG::2014-09-25 17:11:25,259::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`85857448-77b6-4bd1-920b-d17ac56b7c48`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69767::DEBUG::2014-09-25 17:11:25,259::task::993::Storage.TaskManager.Task::(_decref) Task=`85857448-77b6-4bd1-920b-d17ac56b7c48`::ref 1 aborting False >Thread-69767::INFO::2014-09-25 17:11:25,263::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69767::DEBUG::2014-09-25 17:11:25,264::task::1191::Storage.TaskManager.Task::(prepare) Task=`85857448-77b6-4bd1-920b-d17ac56b7c48`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69767::DEBUG::2014-09-25 17:11:25,264::task::595::Storage.TaskManager.Task::(_updateState) Task=`85857448-77b6-4bd1-920b-d17ac56b7c48`::moving from state preparing -> state finished >Thread-69767::DEBUG::2014-09-25 17:11:25,264::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69767::DEBUG::2014-09-25 17:11:25,264::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69767::DEBUG::2014-09-25 17:11:25,265::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69767::DEBUG::2014-09-25 17:11:25,265::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69767::DEBUG::2014-09-25 17:11:25,265::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69767::DEBUG::2014-09-25 17:11:25,265::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69767::DEBUG::2014-09-25 17:11:25,266::task::993::Storage.TaskManager.Task::(_decref) Task=`85857448-77b6-4bd1-920b-d17ac56b7c48`::ref 0 aborting False >Thread-69767::DEBUG::2014-09-25 17:11:25,266::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69767::DEBUG::2014-09-25 17:11:25,267::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:25,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:25,432::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141149 s, 72.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:26,387::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:26,387::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:26,387::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:26,388::__init__::232::IOProcess::(_processLogs) (107322) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:26,392::__init__::232::IOProcess::(_processLogs) (107322) Queuing response >Thread-81::DEBUG::2014-09-25 17:11:26,393::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:11:26,405::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00172569 s, 215 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:26,406::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:26,407::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:26,407::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:26,407::__init__::232::IOProcess::(_processLogs) (107323) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:26,413::__init__::232::IOProcess::(_processLogs) (107323) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:27,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:27,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0239406 s, 42.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:28,066::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:28,067::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69768::DEBUG::2014-09-25 17:11:28,069::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:29,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:29,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145677 s, 70.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:30,887::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:30,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:30,888::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:30,888::__init__::232::IOProcess::(_processLogs) (107324) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:30,891::__init__::232::IOProcess::(_processLogs) (107324) Queuing response >Thread-79::DEBUG::2014-09-25 17:11:30,892::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:11:30,908::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00531652 s, 106 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:30,909::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:30,910::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:30,910::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:30,910::__init__::232::IOProcess::(_processLogs) (107325) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:30,912::__init__::232::IOProcess::(_processLogs) (107325) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:30,913::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:30,914::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:30,915::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:30,915::__init__::232::IOProcess::(_processLogs) (107326) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:30,915::__init__::232::IOProcess::(_processLogs) (107326) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:30,915::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:30,916::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:30,917::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:30,917::__init__::232::IOProcess::(_processLogs) (107327) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:30,917::__init__::232::IOProcess::(_processLogs) (107327) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:30,918::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:30,918::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:30,919::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:30,919::__init__::232::IOProcess::(_processLogs) (107328) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:30,919::__init__::232::IOProcess::(_processLogs) (107328) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:31,083::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:31,084::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69769::DEBUG::2014-09-25 17:11:31,087::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e84477e-9f5b-4e7e-a6d5-cb76b21e3701`::moving from state init -> state preparing >Thread-69769::INFO::2014-09-25 17:11:31,087::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69769::INFO::2014-09-25 17:11:31,088::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00531652', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000432201', 'lastCheck': '8.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00172569', 'lastCheck': '4.7', 'valid': True}} >Thread-69769::DEBUG::2014-09-25 17:11:31,088::task::1191::Storage.TaskManager.Task::(prepare) Task=`5e84477e-9f5b-4e7e-a6d5-cb76b21e3701`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00531652', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000432201', 'lastCheck': '8.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00172569', 'lastCheck': '4.7', 'valid': True}} >Thread-69769::DEBUG::2014-09-25 17:11:31,088::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e84477e-9f5b-4e7e-a6d5-cb76b21e3701`::moving from state preparing -> state finished >Thread-69769::DEBUG::2014-09-25 17:11:31,088::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69769::DEBUG::2014-09-25 17:11:31,088::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69769::DEBUG::2014-09-25 17:11:31,089::task::993::Storage.TaskManager.Task::(_decref) Task=`5e84477e-9f5b-4e7e-a6d5-cb76b21e3701`::ref 0 aborting False >Thread-69769::DEBUG::2014-09-25 17:11:31,091::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:31,108::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:31,110::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69770::DEBUG::2014-09-25 17:11:31,111::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:31,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:31,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.027388 s, 37.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:32,281::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:32,281::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:32,282::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:11:32,283::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:11:32,283::__init__::232::IOProcess::(_processLogs) (107329) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:32,292::__init__::232::IOProcess::(_processLogs) (107329) Queuing response >Thread-80::DEBUG::2014-09-25 17:11:32,293::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000504109 s, 750 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:32,294::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:32,294::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:32,295::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:32,296::__init__::232::IOProcess::(_processLogs) (107330) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:32,296::__init__::232::IOProcess::(_processLogs) (107330) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:33,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:33,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158541 s, 64.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:34,157::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:34,158::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69771::DEBUG::2014-09-25 17:11:34,160::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:35,288::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:35,289::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69772::DEBUG::2014-09-25 17:11:35,289::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69772::DEBUG::2014-09-25 17:11:35,291::task::595::Storage.TaskManager.Task::(_updateState) Task=`04d9bfc9-c6c0-47b6-b1f9-20d1d3d745ab`::moving from state init -> state preparing >Thread-69772::INFO::2014-09-25 17:11:35,292::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69772::INFO::2014-09-25 17:11:35,292::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69772::DEBUG::2014-09-25 17:11:35,292::task::1191::Storage.TaskManager.Task::(prepare) Task=`04d9bfc9-c6c0-47b6-b1f9-20d1d3d745ab`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69772::DEBUG::2014-09-25 17:11:35,293::task::595::Storage.TaskManager.Task::(_updateState) Task=`04d9bfc9-c6c0-47b6-b1f9-20d1d3d745ab`::moving from state preparing -> state finished >Thread-69772::DEBUG::2014-09-25 17:11:35,293::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69772::DEBUG::2014-09-25 17:11:35,293::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69772::DEBUG::2014-09-25 17:11:35,293::task::993::Storage.TaskManager.Task::(_decref) Task=`04d9bfc9-c6c0-47b6-b1f9-20d1d3d745ab`::ref 0 aborting False >Thread-69772::DEBUG::2014-09-25 17:11:35,294::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69772::DEBUG::2014-09-25 17:11:35,294::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:35,304::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69773::DEBUG::2014-09-25 17:11:35,305::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:11:35,305::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69773::DEBUG::2014-09-25 17:11:35,307::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef1d551d-65b9-4a1a-9b76-84322cd0e931`::moving from state init -> state preparing >Thread-69773::INFO::2014-09-25 17:11:35,307::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69773::DEBUG::2014-09-25 17:11:35,309::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cda4536a-1ea1-4a60-979e-cc944dac517b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69773::DEBUG::2014-09-25 17:11:35,309::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69773::DEBUG::2014-09-25 17:11:35,310::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69773::DEBUG::2014-09-25 17:11:35,310::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cda4536a-1ea1-4a60-979e-cc944dac517b`::Granted request >Thread-69773::DEBUG::2014-09-25 17:11:35,311::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ef1d551d-65b9-4a1a-9b76-84322cd0e931`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69773::DEBUG::2014-09-25 17:11:35,311::task::993::Storage.TaskManager.Task::(_decref) Task=`ef1d551d-65b9-4a1a-9b76-84322cd0e931`::ref 1 aborting False >Thread-69773::INFO::2014-09-25 17:11:35,315::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69773::DEBUG::2014-09-25 17:11:35,316::task::1191::Storage.TaskManager.Task::(prepare) Task=`ef1d551d-65b9-4a1a-9b76-84322cd0e931`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69773::DEBUG::2014-09-25 17:11:35,316::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef1d551d-65b9-4a1a-9b76-84322cd0e931`::moving from state preparing -> state finished >Thread-69773::DEBUG::2014-09-25 17:11:35,316::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69773::DEBUG::2014-09-25 17:11:35,316::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69773::DEBUG::2014-09-25 17:11:35,317::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69773::DEBUG::2014-09-25 17:11:35,317::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69773::DEBUG::2014-09-25 17:11:35,317::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69773::DEBUG::2014-09-25 17:11:35,317::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69773::DEBUG::2014-09-25 17:11:35,318::task::993::Storage.TaskManager.Task::(_decref) Task=`ef1d551d-65b9-4a1a-9b76-84322cd0e931`::ref 0 aborting False >Thread-69773::DEBUG::2014-09-25 17:11:35,318::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69773::DEBUG::2014-09-25 17:11:35,319::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:35,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:35,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0226222 s, 45.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:36,415::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:36,416::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:36,416::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:36,416::__init__::232::IOProcess::(_processLogs) (107331) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:36,417::__init__::232::IOProcess::(_processLogs) (107331) Queuing response >Thread-81::DEBUG::2014-09-25 17:11:36,418::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:11:36,427::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000377961 s, 982 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:36,428::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:36,428::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:36,429::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:36,429::__init__::232::IOProcess::(_processLogs) (107332) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:36,429::__init__::232::IOProcess::(_processLogs) (107332) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:37,177::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:37,179::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69774::DEBUG::2014-09-25 17:11:37,181::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:37,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:37,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0271263 s, 37.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:11:39,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:39,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212973 s, 48.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:40,196::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:40,197::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69775::DEBUG::2014-09-25 17:11:40,200::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:11:40,920::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:40,920::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:40,920::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:40,920::__init__::232::IOProcess::(_processLogs) (107333) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:40,924::__init__::232::IOProcess::(_processLogs) (107333) Queuing response >Thread-79::DEBUG::2014-09-25 17:11:40,925::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:11:40,938::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00336585 s, 167 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:40,939::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:40,940::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:40,940::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:40,940::__init__::232::IOProcess::(_processLogs) (107334) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:40,943::__init__::232::IOProcess::(_processLogs) (107334) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:40,945::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:40,945::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:40,945::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:40,945::__init__::232::IOProcess::(_processLogs) (107335) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:40,949::__init__::232::IOProcess::(_processLogs) (107335) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:40,950::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:40,950::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:40,950::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:40,950::__init__::232::IOProcess::(_processLogs) (107336) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:40,952::__init__::232::IOProcess::(_processLogs) (107336) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:40,953::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:40,953::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:40,953::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:40,953::__init__::232::IOProcess::(_processLogs) (107337) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:40,958::__init__::232::IOProcess::(_processLogs) (107337) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:41,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:41,877::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215374 s, 47.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:42,297::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:42,297::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:42,298::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:42,298::__init__::232::IOProcess::(_processLogs) (107338) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:42,298::__init__::232::IOProcess::(_processLogs) (107338) Queuing response >Thread-80::DEBUG::2014-09-25 17:11:42,300::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:11:42,309::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000405783 s, 932 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:42,310::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:42,311::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:42,311::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:42,311::__init__::232::IOProcess::(_processLogs) (107339) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:42,312::__init__::232::IOProcess::(_processLogs) (107339) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:43,212::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:43,213::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69776::DEBUG::2014-09-25 17:11:43,216::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:43,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:43,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179055 s, 57.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:45,343::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69777::DEBUG::2014-09-25 17:11:45,345::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:11:45,345::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69777::DEBUG::2014-09-25 17:11:45,347::task::595::Storage.TaskManager.Task::(_updateState) Task=`76c39531-c6e1-4174-acf9-eab9e18f1b3c`::moving from state init -> state preparing >Thread-69777::INFO::2014-09-25 17:11:45,348::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69777::INFO::2014-09-25 17:11:45,348::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69777::DEBUG::2014-09-25 17:11:45,349::task::1191::Storage.TaskManager.Task::(prepare) Task=`76c39531-c6e1-4174-acf9-eab9e18f1b3c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69777::DEBUG::2014-09-25 17:11:45,349::task::595::Storage.TaskManager.Task::(_updateState) Task=`76c39531-c6e1-4174-acf9-eab9e18f1b3c`::moving from state preparing -> state finished >Thread-69777::DEBUG::2014-09-25 17:11:45,349::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69777::DEBUG::2014-09-25 17:11:45,349::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69777::DEBUG::2014-09-25 17:11:45,350::task::993::Storage.TaskManager.Task::(_decref) Task=`76c39531-c6e1-4174-acf9-eab9e18f1b3c`::ref 0 aborting False >Thread-69777::DEBUG::2014-09-25 17:11:45,350::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69777::DEBUG::2014-09-25 17:11:45,350::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:45,363::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:45,365::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69778::DEBUG::2014-09-25 17:11:45,365::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69778::DEBUG::2014-09-25 17:11:45,367::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b3fe00a-ef67-4bec-bd34-98bc2cad770d`::moving from state init -> state preparing >Thread-69778::INFO::2014-09-25 17:11:45,367::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69778::DEBUG::2014-09-25 17:11:45,368::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`26afa9e5-7ad9-4eff-9d1b-3c48a3276c30`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69778::DEBUG::2014-09-25 17:11:45,368::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69778::DEBUG::2014-09-25 17:11:45,368::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69778::DEBUG::2014-09-25 17:11:45,369::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`26afa9e5-7ad9-4eff-9d1b-3c48a3276c30`::Granted request >Thread-69778::DEBUG::2014-09-25 17:11:45,369::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1b3fe00a-ef67-4bec-bd34-98bc2cad770d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69778::DEBUG::2014-09-25 17:11:45,370::task::993::Storage.TaskManager.Task::(_decref) Task=`1b3fe00a-ef67-4bec-bd34-98bc2cad770d`::ref 1 aborting False >Thread-69778::INFO::2014-09-25 17:11:45,375::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69778::DEBUG::2014-09-25 17:11:45,375::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b3fe00a-ef67-4bec-bd34-98bc2cad770d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69778::DEBUG::2014-09-25 17:11:45,375::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b3fe00a-ef67-4bec-bd34-98bc2cad770d`::moving from state preparing -> state finished >Thread-69778::DEBUG::2014-09-25 17:11:45,375::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69778::DEBUG::2014-09-25 17:11:45,376::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69778::DEBUG::2014-09-25 17:11:45,376::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69778::DEBUG::2014-09-25 17:11:45,376::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69778::DEBUG::2014-09-25 17:11:45,377::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69778::DEBUG::2014-09-25 17:11:45,377::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69778::DEBUG::2014-09-25 17:11:45,377::task::993::Storage.TaskManager.Task::(_decref) Task=`1b3fe00a-ef67-4bec-bd34-98bc2cad770d`::ref 0 aborting False >Thread-69778::DEBUG::2014-09-25 17:11:45,378::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69778::DEBUG::2014-09-25 17:11:45,378::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:45,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:45,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169809 s, 60.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:46,227::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:46,232::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69779::DEBUG::2014-09-25 17:11:46,239::task::595::Storage.TaskManager.Task::(_updateState) Task=`9280ee1b-4db9-4ec2-9cdd-54192a1e3d04`::moving from state init -> state preparing >Thread-69779::INFO::2014-09-25 17:11:46,240::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69779::INFO::2014-09-25 17:11:46,242::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00336585', 'lastCheck': '5.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000405783', 'lastCheck': '3.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000377961', 'lastCheck': '9.8', 'valid': True}} >Thread-69779::DEBUG::2014-09-25 17:11:46,243::task::1191::Storage.TaskManager.Task::(prepare) Task=`9280ee1b-4db9-4ec2-9cdd-54192a1e3d04`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00336585', 'lastCheck': '5.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000405783', 'lastCheck': '3.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000377961', 'lastCheck': '9.8', 'valid': True}} >Thread-69779::DEBUG::2014-09-25 17:11:46,244::task::595::Storage.TaskManager.Task::(_updateState) Task=`9280ee1b-4db9-4ec2-9cdd-54192a1e3d04`::moving from state preparing -> state finished >Thread-69779::DEBUG::2014-09-25 17:11:46,245::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69779::DEBUG::2014-09-25 17:11:46,245::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69779::DEBUG::2014-09-25 17:11:46,246::task::993::Storage.TaskManager.Task::(_decref) Task=`9280ee1b-4db9-4ec2-9cdd-54192a1e3d04`::ref 0 aborting False >Thread-69779::DEBUG::2014-09-25 17:11:46,254::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:46,273::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:46,276::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69780::DEBUG::2014-09-25 17:11:46,281::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:11:46,432::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:46,433::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:46,434::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:46,435::__init__::232::IOProcess::(_processLogs) (107340) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:46,437::__init__::232::IOProcess::(_processLogs) (107340) Queuing response >Thread-81::DEBUG::2014-09-25 17:11:46,439::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:11:46,457::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00192846 s, 192 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:46,459::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:46,461::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:46,462::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:46,463::__init__::232::IOProcess::(_processLogs) (107341) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:46,464::__init__::232::IOProcess::(_processLogs) (107341) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:47,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:48,019::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133397 s, 76.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:49,327::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:49,329::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69781::DEBUG::2014-09-25 17:11:49,331::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:50,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:50,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168822 s, 60.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:50,959::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:50,960::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:50,960::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:50,960::__init__::232::IOProcess::(_processLogs) (107342) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:50,965::__init__::232::IOProcess::(_processLogs) (107342) Queuing response >Thread-79::DEBUG::2014-09-25 17:11:50,966::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:11:50,982::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00253143 s, 222 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:50,984::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:50,984::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:50,984::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:50,984::__init__::232::IOProcess::(_processLogs) (107343) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:50,991::__init__::232::IOProcess::(_processLogs) (107343) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:50,992::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:50,992::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:50,993::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:50,993::__init__::232::IOProcess::(_processLogs) (107344) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:50,994::__init__::232::IOProcess::(_processLogs) (107344) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:50,994::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:50,994::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:50,995::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:50,996::__init__::232::IOProcess::(_processLogs) (107345) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:50,996::__init__::232::IOProcess::(_processLogs) (107345) Queuing response >Thread-29::DEBUG::2014-09-25 17:11:50,996::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:50,997::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:50,998::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:50,998::__init__::232::IOProcess::(_processLogs) (107346) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:11:50,998::__init__::232::IOProcess::(_processLogs) (107346) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:52,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:52,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134366 s, 76.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:52,314::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:52,314::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:52,315::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:52,315::__init__::232::IOProcess::(_processLogs) (107347) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:52,315::__init__::232::IOProcess::(_processLogs) (107347) Queuing response >Thread-80::DEBUG::2014-09-25 17:11:52,317::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:11:52,325::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00043075 s, 878 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:52,327::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:52,327::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:52,328::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:52,329::__init__::232::IOProcess::(_processLogs) (107348) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:52,329::__init__::232::IOProcess::(_processLogs) (107348) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:52,345::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:52,346::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69782::DEBUG::2014-09-25 17:11:52,348::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:54,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:54,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014387 s, 71.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:55,365::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:55,367::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69783::DEBUG::2014-09-25 17:11:55,368::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:55,409::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69784::DEBUG::2014-09-25 17:11:55,410::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:11:55,410::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69784::DEBUG::2014-09-25 17:11:55,412::task::595::Storage.TaskManager.Task::(_updateState) Task=`2455c26e-46aa-4435-b0e8-099cec088701`::moving from state init -> state preparing >Thread-69784::INFO::2014-09-25 17:11:55,413::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69784::INFO::2014-09-25 17:11:55,413::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69784::DEBUG::2014-09-25 17:11:55,413::task::1191::Storage.TaskManager.Task::(prepare) Task=`2455c26e-46aa-4435-b0e8-099cec088701`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69784::DEBUG::2014-09-25 17:11:55,414::task::595::Storage.TaskManager.Task::(_updateState) Task=`2455c26e-46aa-4435-b0e8-099cec088701`::moving from state preparing -> state finished >Thread-69784::DEBUG::2014-09-25 17:11:55,414::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69784::DEBUG::2014-09-25 17:11:55,414::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69784::DEBUG::2014-09-25 17:11:55,414::task::993::Storage.TaskManager.Task::(_decref) Task=`2455c26e-46aa-4435-b0e8-099cec088701`::ref 0 aborting False >Thread-69784::DEBUG::2014-09-25 17:11:55,415::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69784::DEBUG::2014-09-25 17:11:55,415::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:55,429::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69785::DEBUG::2014-09-25 17:11:55,431::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:11:55,431::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69785::DEBUG::2014-09-25 17:11:55,433::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ae00480-2c5f-402b-a2d4-ddf2ac237c28`::moving from state init -> state preparing >Thread-69785::INFO::2014-09-25 17:11:55,433::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69785::DEBUG::2014-09-25 17:11:55,434::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`459536be-ba7e-48fe-a5ee-f5d65df11eb2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69785::DEBUG::2014-09-25 17:11:55,434::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69785::DEBUG::2014-09-25 17:11:55,434::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69785::DEBUG::2014-09-25 17:11:55,435::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`459536be-ba7e-48fe-a5ee-f5d65df11eb2`::Granted request >Thread-69785::DEBUG::2014-09-25 17:11:55,435::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3ae00480-2c5f-402b-a2d4-ddf2ac237c28`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69785::DEBUG::2014-09-25 17:11:55,435::task::993::Storage.TaskManager.Task::(_decref) Task=`3ae00480-2c5f-402b-a2d4-ddf2ac237c28`::ref 1 aborting False >Thread-69785::INFO::2014-09-25 17:11:55,440::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69785::DEBUG::2014-09-25 17:11:55,440::task::1191::Storage.TaskManager.Task::(prepare) Task=`3ae00480-2c5f-402b-a2d4-ddf2ac237c28`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69785::DEBUG::2014-09-25 17:11:55,440::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ae00480-2c5f-402b-a2d4-ddf2ac237c28`::moving from state preparing -> state finished >Thread-69785::DEBUG::2014-09-25 17:11:55,441::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69785::DEBUG::2014-09-25 17:11:55,441::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69785::DEBUG::2014-09-25 17:11:55,441::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69785::DEBUG::2014-09-25 17:11:55,442::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69785::DEBUG::2014-09-25 17:11:55,442::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69785::DEBUG::2014-09-25 17:11:55,442::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69785::DEBUG::2014-09-25 17:11:55,442::task::993::Storage.TaskManager.Task::(_decref) Task=`3ae00480-2c5f-402b-a2d4-ddf2ac237c28`::ref 0 aborting False >Thread-69785::DEBUG::2014-09-25 17:11:55,443::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69785::DEBUG::2014-09-25 17:11:55,443::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:11:56,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:56,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133087 s, 76.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:56,468::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:56,468::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:56,468::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:56,469::__init__::232::IOProcess::(_processLogs) (107349) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:56,481::__init__::232::IOProcess::(_processLogs) (107349) Queuing response >Thread-81::DEBUG::2014-09-25 17:11:56,483::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:11:56,525::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00051792 s, 716 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:11:56,526::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:11:56,527::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:11:56,527::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:11:56,527::__init__::232::IOProcess::(_processLogs) (107350) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:11:56,536::__init__::232::IOProcess::(_processLogs) (107350) Queuing response >Dummy-148::DEBUG::2014-09-25 17:11:58,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:11:58,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135078 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:11:58,384::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:11:58,386::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69786::DEBUG::2014-09-25 17:11:58,388::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:00,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:00,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170497 s, 60.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:00,999::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:00,999::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:00,999::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:01,000::__init__::232::IOProcess::(_processLogs) (107351) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:01,000::__init__::232::IOProcess::(_processLogs) (107351) Queuing response >Thread-79::DEBUG::2014-09-25 17:12:01,002::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:12:01,020::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00302456 s, 186 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:01,021::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:01,022::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:01,022::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:01,022::__init__::232::IOProcess::(_processLogs) (107352) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:01,027::__init__::232::IOProcess::(_processLogs) (107352) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:01,028::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:01,029::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:01,029::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:01,029::__init__::232::IOProcess::(_processLogs) (107353) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:01,029::__init__::232::IOProcess::(_processLogs) (107353) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:01,030::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:01,031::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:01,032::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:01,032::__init__::232::IOProcess::(_processLogs) (107354) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:01,032::__init__::232::IOProcess::(_processLogs) (107354) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:01,032::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:01,033::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:01,034::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:01,034::__init__::232::IOProcess::(_processLogs) (107355) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:01,034::__init__::232::IOProcess::(_processLogs) (107355) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:01,401::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:01,403::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69787::DEBUG::2014-09-25 17:12:01,405::task::595::Storage.TaskManager.Task::(_updateState) Task=`502cc60a-6b5a-48b2-bb73-6c7d195cfc44`::moving from state init -> state preparing >Thread-69787::INFO::2014-09-25 17:12:01,406::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69787::INFO::2014-09-25 17:12:01,406::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00302456', 'lastCheck': '0.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00043075', 'lastCheck': '9.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00051792', 'lastCheck': '4.9', 'valid': True}} >Thread-69787::DEBUG::2014-09-25 17:12:01,406::task::1191::Storage.TaskManager.Task::(prepare) Task=`502cc60a-6b5a-48b2-bb73-6c7d195cfc44`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00302456', 'lastCheck': '0.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00043075', 'lastCheck': '9.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00051792', 'lastCheck': '4.9', 'valid': True}} >Thread-69787::DEBUG::2014-09-25 17:12:01,407::task::595::Storage.TaskManager.Task::(_updateState) Task=`502cc60a-6b5a-48b2-bb73-6c7d195cfc44`::moving from state preparing -> state finished >Thread-69787::DEBUG::2014-09-25 17:12:01,407::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69787::DEBUG::2014-09-25 17:12:01,407::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69787::DEBUG::2014-09-25 17:12:01,407::task::993::Storage.TaskManager.Task::(_decref) Task=`502cc60a-6b5a-48b2-bb73-6c7d195cfc44`::ref 0 aborting False >Thread-69787::DEBUG::2014-09-25 17:12:01,410::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:01,429::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:01,430::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69788::DEBUG::2014-09-25 17:12:01,432::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:02,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:12:02,331::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:02,332::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:02,334::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:02,335::__init__::232::IOProcess::(_processLogs) (107356) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:02,337::__init__::232::IOProcess::(_processLogs) (107356) Queuing response >Thread-80::DEBUG::2014-09-25 17:12:02,339::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:12:02,349::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000351797 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:02,352::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:02,353::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:02,355::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:02,356::__init__::232::IOProcess::(_processLogs) (107357) Got request for method 'statvfs' >Dummy-148::DEBUG::2014-09-25 17:12:02,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0343253 s, 29.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:02,358::__init__::232::IOProcess::(_processLogs) (107357) Queuing response >Dummy-148::DEBUG::2014-09-25 17:12:04,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:04,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136142 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:04,475::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:04,477::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69789::DEBUG::2014-09-25 17:12:04,480::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:05,478::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69790::DEBUG::2014-09-25 17:12:05,479::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:12:05,479::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69790::DEBUG::2014-09-25 17:12:05,481::task::595::Storage.TaskManager.Task::(_updateState) Task=`614c622c-5e00-42d6-8d1a-6a81594fcc0f`::moving from state init -> state preparing >Thread-69790::INFO::2014-09-25 17:12:05,482::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69790::INFO::2014-09-25 17:12:05,482::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69790::DEBUG::2014-09-25 17:12:05,483::task::1191::Storage.TaskManager.Task::(prepare) Task=`614c622c-5e00-42d6-8d1a-6a81594fcc0f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69790::DEBUG::2014-09-25 17:12:05,483::task::595::Storage.TaskManager.Task::(_updateState) Task=`614c622c-5e00-42d6-8d1a-6a81594fcc0f`::moving from state preparing -> state finished >Thread-69790::DEBUG::2014-09-25 17:12:05,483::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69790::DEBUG::2014-09-25 17:12:05,483::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69790::DEBUG::2014-09-25 17:12:05,483::task::993::Storage.TaskManager.Task::(_decref) Task=`614c622c-5e00-42d6-8d1a-6a81594fcc0f`::ref 0 aborting False >Thread-69790::DEBUG::2014-09-25 17:12:05,484::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69790::DEBUG::2014-09-25 17:12:05,484::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:05,496::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:05,498::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69791::DEBUG::2014-09-25 17:12:05,498::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69791::DEBUG::2014-09-25 17:12:05,499::task::595::Storage.TaskManager.Task::(_updateState) Task=`f029f17d-2142-4fb7-9233-a442aff95a54`::moving from state init -> state preparing >Thread-69791::INFO::2014-09-25 17:12:05,500::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69791::DEBUG::2014-09-25 17:12:05,500::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0b8fffd8-7042-4a27-a3a5-ed2b0da91ce5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69791::DEBUG::2014-09-25 17:12:05,501::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69791::DEBUG::2014-09-25 17:12:05,501::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69791::DEBUG::2014-09-25 17:12:05,501::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0b8fffd8-7042-4a27-a3a5-ed2b0da91ce5`::Granted request >Thread-69791::DEBUG::2014-09-25 17:12:05,502::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f029f17d-2142-4fb7-9233-a442aff95a54`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69791::DEBUG::2014-09-25 17:12:05,502::task::993::Storage.TaskManager.Task::(_decref) Task=`f029f17d-2142-4fb7-9233-a442aff95a54`::ref 1 aborting False >Thread-69791::INFO::2014-09-25 17:12:05,506::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69791::DEBUG::2014-09-25 17:12:05,507::task::1191::Storage.TaskManager.Task::(prepare) Task=`f029f17d-2142-4fb7-9233-a442aff95a54`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69791::DEBUG::2014-09-25 17:12:05,507::task::595::Storage.TaskManager.Task::(_updateState) Task=`f029f17d-2142-4fb7-9233-a442aff95a54`::moving from state preparing -> state finished >Thread-69791::DEBUG::2014-09-25 17:12:05,507::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69791::DEBUG::2014-09-25 17:12:05,507::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69791::DEBUG::2014-09-25 17:12:05,508::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69791::DEBUG::2014-09-25 17:12:05,508::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69791::DEBUG::2014-09-25 17:12:05,508::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69791::DEBUG::2014-09-25 17:12:05,509::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69791::DEBUG::2014-09-25 17:12:05,509::task::993::Storage.TaskManager.Task::(_decref) Task=`f029f17d-2142-4fb7-9233-a442aff95a54`::ref 0 aborting False >Thread-69791::DEBUG::2014-09-25 17:12:05,509::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69791::DEBUG::2014-09-25 17:12:05,510::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:06,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:06,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156545 s, 65.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:06,539::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:06,539::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:06,539::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:06,540::__init__::232::IOProcess::(_processLogs) (107358) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:06,543::__init__::232::IOProcess::(_processLogs) (107358) Queuing response >Thread-81::DEBUG::2014-09-25 17:12:06,544::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:12:06,565::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00764411 s, 48.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:06,566::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:06,566::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:06,566::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:06,567::__init__::232::IOProcess::(_processLogs) (107359) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:06,586::__init__::232::IOProcess::(_processLogs) (107359) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:07,493::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:07,494::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69792::DEBUG::2014-09-25 17:12:07,496::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:08,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:08,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192366 s, 53.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:10,508::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:10,509::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69793::DEBUG::2014-09-25 17:12:10,511::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:10,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:10,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0254046 s, 40.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:11,035::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:11,035::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:11,035::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:11,036::__init__::232::IOProcess::(_processLogs) (107360) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:11,036::__init__::232::IOProcess::(_processLogs) (107360) Queuing response >Thread-79::DEBUG::2014-09-25 17:12:11,037::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:12:11,046::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000419046 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:11,048::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:11,048::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:11,048::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:11,049::__init__::232::IOProcess::(_processLogs) (107361) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:11,052::__init__::232::IOProcess::(_processLogs) (107361) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:11,053::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:11,053::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:11,054::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:11,054::__init__::232::IOProcess::(_processLogs) (107362) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:11,054::__init__::232::IOProcess::(_processLogs) (107362) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:11,055::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:11,055::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:11,055::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:11,056::__init__::232::IOProcess::(_processLogs) (107363) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:11,056::__init__::232::IOProcess::(_processLogs) (107363) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:11,057::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:11,057::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:11,058::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:11,059::__init__::232::IOProcess::(_processLogs) (107364) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:11,059::__init__::232::IOProcess::(_processLogs) (107364) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:12,360::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:12,361::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:12,361::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:12,361::__init__::232::IOProcess::(_processLogs) (107365) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:12,362::__init__::232::IOProcess::(_processLogs) (107365) Queuing response >Thread-80::DEBUG::2014-09-25 17:12:12,363::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:12:12,374::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00185413 s, 204 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:12,375::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:12,376::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:12,376::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:12,376::__init__::232::IOProcess::(_processLogs) (107366) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:12,377::__init__::232::IOProcess::(_processLogs) (107366) Queuing response >Dummy-148::DEBUG::2014-09-25 17:12:12,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:12,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133472 s, 76.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:13,524::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:13,526::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69794::DEBUG::2014-09-25 17:12:13,528::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:14,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:14,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134933 s, 75.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:15,536::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:15,537::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69795::DEBUG::2014-09-25 17:12:15,537::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69795::DEBUG::2014-09-25 17:12:15,540::task::595::Storage.TaskManager.Task::(_updateState) Task=`75ace538-f896-433d-8a11-2b5b9a2cd6ee`::moving from state init -> state preparing >Thread-69795::INFO::2014-09-25 17:12:15,540::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69795::INFO::2014-09-25 17:12:15,541::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69795::DEBUG::2014-09-25 17:12:15,541::task::1191::Storage.TaskManager.Task::(prepare) Task=`75ace538-f896-433d-8a11-2b5b9a2cd6ee`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69795::DEBUG::2014-09-25 17:12:15,541::task::595::Storage.TaskManager.Task::(_updateState) Task=`75ace538-f896-433d-8a11-2b5b9a2cd6ee`::moving from state preparing -> state finished >Thread-69795::DEBUG::2014-09-25 17:12:15,541::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69795::DEBUG::2014-09-25 17:12:15,542::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69795::DEBUG::2014-09-25 17:12:15,542::task::993::Storage.TaskManager.Task::(_decref) Task=`75ace538-f896-433d-8a11-2b5b9a2cd6ee`::ref 0 aborting False >Thread-69795::DEBUG::2014-09-25 17:12:15,542::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69795::DEBUG::2014-09-25 17:12:15,543::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:15,554::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:15,555::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69796::DEBUG::2014-09-25 17:12:15,555::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69796::DEBUG::2014-09-25 17:12:15,557::task::595::Storage.TaskManager.Task::(_updateState) Task=`f20bf394-4955-4b54-88b7-829710c6ca9a`::moving from state init -> state preparing >Thread-69796::INFO::2014-09-25 17:12:15,557::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69796::DEBUG::2014-09-25 17:12:15,558::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e02445e2-d5ea-4272-9ddf-190e1588ce1c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69796::DEBUG::2014-09-25 17:12:15,558::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69796::DEBUG::2014-09-25 17:12:15,558::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69796::DEBUG::2014-09-25 17:12:15,559::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e02445e2-d5ea-4272-9ddf-190e1588ce1c`::Granted request >Thread-69796::DEBUG::2014-09-25 17:12:15,559::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f20bf394-4955-4b54-88b7-829710c6ca9a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69796::DEBUG::2014-09-25 17:12:15,559::task::993::Storage.TaskManager.Task::(_decref) Task=`f20bf394-4955-4b54-88b7-829710c6ca9a`::ref 1 aborting False >Thread-69796::INFO::2014-09-25 17:12:15,564::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69796::DEBUG::2014-09-25 17:12:15,565::task::1191::Storage.TaskManager.Task::(prepare) Task=`f20bf394-4955-4b54-88b7-829710c6ca9a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69796::DEBUG::2014-09-25 17:12:15,565::task::595::Storage.TaskManager.Task::(_updateState) Task=`f20bf394-4955-4b54-88b7-829710c6ca9a`::moving from state preparing -> state finished >Thread-69796::DEBUG::2014-09-25 17:12:15,565::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69796::DEBUG::2014-09-25 17:12:15,565::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69796::DEBUG::2014-09-25 17:12:15,566::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69796::DEBUG::2014-09-25 17:12:15,566::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69796::DEBUG::2014-09-25 17:12:15,566::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69796::DEBUG::2014-09-25 17:12:15,567::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69796::DEBUG::2014-09-25 17:12:15,567::task::993::Storage.TaskManager.Task::(_decref) Task=`f20bf394-4955-4b54-88b7-829710c6ca9a`::ref 0 aborting False >Thread-69796::DEBUG::2014-09-25 17:12:15,567::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69796::DEBUG::2014-09-25 17:12:15,568::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:16,544::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:16,548::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69797::DEBUG::2014-09-25 17:12:16,556::task::595::Storage.TaskManager.Task::(_updateState) Task=`ed928a0a-b84f-492d-b13b-45715cef0bcf`::moving from state init -> state preparing >Thread-69797::INFO::2014-09-25 17:12:16,557::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69797::INFO::2014-09-25 17:12:16,559::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000419046', 'lastCheck': '5.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00185413', 'lastCheck': '4.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00764411', 'lastCheck': '10.0', 'valid': True}} >Thread-69797::DEBUG::2014-09-25 17:12:16,560::task::1191::Storage.TaskManager.Task::(prepare) Task=`ed928a0a-b84f-492d-b13b-45715cef0bcf`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000419046', 'lastCheck': '5.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00185413', 'lastCheck': '4.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00764411', 'lastCheck': '10.0', 'valid': True}} >Thread-69797::DEBUG::2014-09-25 17:12:16,561::task::595::Storage.TaskManager.Task::(_updateState) Task=`ed928a0a-b84f-492d-b13b-45715cef0bcf`::moving from state preparing -> state finished >Thread-69797::DEBUG::2014-09-25 17:12:16,562::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69797::DEBUG::2014-09-25 17:12:16,563::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69797::DEBUG::2014-09-25 17:12:16,564::task::993::Storage.TaskManager.Task::(_decref) Task=`ed928a0a-b84f-492d-b13b-45715cef0bcf`::ref 0 aborting False >Thread-69797::DEBUG::2014-09-25 17:12:16,573::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:12:16,589::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:16,590::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:16,592::__init__::232::IOProcess::(_processLogs) Extracting request information... >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:16,592::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-29::DEBUG::2014-09-25 17:12:16,593::__init__::232::IOProcess::(_processLogs) (107367) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:16,597::__init__::232::IOProcess::(_processLogs) (107367) Queuing response >JsonRpcServer::DEBUG::2014-09-25 17:12:16,598::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-81::DEBUG::2014-09-25 17:12:16,601::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-69798::DEBUG::2014-09-25 17:12:16,615::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 17:12:16,625::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00201439 s, 184 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:16,628::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:16,629::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:16,630::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:16,631::__init__::232::IOProcess::(_processLogs) (107368) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:16,633::__init__::232::IOProcess::(_processLogs) (107368) Queuing response >Dummy-148::DEBUG::2014-09-25 17:12:16,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:16,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.027504 s, 37.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:12:18,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:18,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0195525 s, 52.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:19,668::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:19,669::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69799::DEBUG::2014-09-25 17:12:19,671::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:20,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:20,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201719 s, 50.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:21,059::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:21,060::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:21,060::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:21,060::__init__::232::IOProcess::(_processLogs) (107369) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:21,063::__init__::232::IOProcess::(_processLogs) (107369) Queuing response >Thread-79::DEBUG::2014-09-25 17:12:21,064::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:12:21,080::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00443116 s, 127 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:21,082::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:21,082::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:21,082::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:21,083::__init__::232::IOProcess::(_processLogs) (107370) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:21,087::__init__::232::IOProcess::(_processLogs) (107370) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:21,088::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:21,089::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:21,089::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:21,090::__init__::232::IOProcess::(_processLogs) (107371) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:21,090::__init__::232::IOProcess::(_processLogs) (107371) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:21,090::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:21,091::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:21,091::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:21,092::__init__::232::IOProcess::(_processLogs) (107372) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:21,092::__init__::232::IOProcess::(_processLogs) (107372) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:21,092::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:21,093::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:21,094::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:21,094::__init__::232::IOProcess::(_processLogs) (107373) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:21,094::__init__::232::IOProcess::(_processLogs) (107373) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:22,379::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:22,380::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:22,380::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:22,380::__init__::232::IOProcess::(_processLogs) (107374) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:22,380::__init__::232::IOProcess::(_processLogs) (107374) Queuing response >Thread-80::DEBUG::2014-09-25 17:12:22,382::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:12:22,391::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000533169 s, 709 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:22,392::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:22,392::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:22,393::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:22,395::__init__::232::IOProcess::(_processLogs) (107375) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:22,395::__init__::232::IOProcess::(_processLogs) (107375) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:22,688::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:22,690::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69800::DEBUG::2014-09-25 17:12:22,692::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:22,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:23,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181969 s, 56.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:12:25,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:25,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161437 s, 63.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:25,594::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:25,596::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69801::DEBUG::2014-09-25 17:12:25,596::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69801::DEBUG::2014-09-25 17:12:25,598::task::595::Storage.TaskManager.Task::(_updateState) Task=`59fd15c6-da28-4453-a91c-e944b4e66c69`::moving from state init -> state preparing >Thread-69801::INFO::2014-09-25 17:12:25,598::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69801::INFO::2014-09-25 17:12:25,599::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69801::DEBUG::2014-09-25 17:12:25,599::task::1191::Storage.TaskManager.Task::(prepare) Task=`59fd15c6-da28-4453-a91c-e944b4e66c69`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69801::DEBUG::2014-09-25 17:12:25,599::task::595::Storage.TaskManager.Task::(_updateState) Task=`59fd15c6-da28-4453-a91c-e944b4e66c69`::moving from state preparing -> state finished >Thread-69801::DEBUG::2014-09-25 17:12:25,600::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69801::DEBUG::2014-09-25 17:12:25,600::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69801::DEBUG::2014-09-25 17:12:25,600::task::993::Storage.TaskManager.Task::(_decref) Task=`59fd15c6-da28-4453-a91c-e944b4e66c69`::ref 0 aborting False >Thread-69801::DEBUG::2014-09-25 17:12:25,600::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69801::DEBUG::2014-09-25 17:12:25,601::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:25,617::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69802::DEBUG::2014-09-25 17:12:25,618::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:12:25,618::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69802::DEBUG::2014-09-25 17:12:25,620::task::595::Storage.TaskManager.Task::(_updateState) Task=`19d88416-f588-4c79-b6a6-3f9d1f79c3fd`::moving from state init -> state preparing >Thread-69802::INFO::2014-09-25 17:12:25,621::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69802::DEBUG::2014-09-25 17:12:25,621::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`30f0a3b9-95a7-4e94-9590-ec81ff0a86d0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69802::DEBUG::2014-09-25 17:12:25,621::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69802::DEBUG::2014-09-25 17:12:25,622::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69802::DEBUG::2014-09-25 17:12:25,622::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`30f0a3b9-95a7-4e94-9590-ec81ff0a86d0`::Granted request >Thread-69802::DEBUG::2014-09-25 17:12:25,623::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`19d88416-f588-4c79-b6a6-3f9d1f79c3fd`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69802::DEBUG::2014-09-25 17:12:25,623::task::993::Storage.TaskManager.Task::(_decref) Task=`19d88416-f588-4c79-b6a6-3f9d1f79c3fd`::ref 1 aborting False >Thread-69802::INFO::2014-09-25 17:12:25,627::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69802::DEBUG::2014-09-25 17:12:25,628::task::1191::Storage.TaskManager.Task::(prepare) Task=`19d88416-f588-4c79-b6a6-3f9d1f79c3fd`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69802::DEBUG::2014-09-25 17:12:25,628::task::595::Storage.TaskManager.Task::(_updateState) Task=`19d88416-f588-4c79-b6a6-3f9d1f79c3fd`::moving from state preparing -> state finished >Thread-69802::DEBUG::2014-09-25 17:12:25,628::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69802::DEBUG::2014-09-25 17:12:25,628::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69802::DEBUG::2014-09-25 17:12:25,629::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69802::DEBUG::2014-09-25 17:12:25,629::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69802::DEBUG::2014-09-25 17:12:25,629::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69802::DEBUG::2014-09-25 17:12:25,629::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69802::DEBUG::2014-09-25 17:12:25,630::task::993::Storage.TaskManager.Task::(_decref) Task=`19d88416-f588-4c79-b6a6-3f9d1f79c3fd`::ref 0 aborting False >Thread-69802::DEBUG::2014-09-25 17:12:25,630::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69802::DEBUG::2014-09-25 17:12:25,631::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:25,709::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:25,710::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69803::DEBUG::2014-09-25 17:12:25,712::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 17:12:26,635::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:12:26,639::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:26,640::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:26,640::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:26,640::__init__::232::IOProcess::(_processLogs) (107376) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:12:26,641::__init__::232::IOProcess::(_processLogs) (107376) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:26,643::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:26,644::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:26,646::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:26,647::__init__::232::IOProcess::(_processLogs) (107377) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:12:26,650::__init__::232::IOProcess::(_processLogs) (107377) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:26,653::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:26,654::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:26,655::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:12:26,657::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:12:26,657::__init__::232::IOProcess::(_processLogs) (107378) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:12:26,659::__init__::232::IOProcess::(_processLogs) (107378) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:26,659::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:26,660::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:26,660::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:26,660::__init__::232::IOProcess::(_processLogs) (107379) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:12:26,685::__init__::232::IOProcess::(_processLogs) (107379) Queuing response >Thread-81::DEBUG::2014-09-25 17:12:26,686::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:12:26,687::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:26,687::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:26,687::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:26,688::__init__::232::IOProcess::(_processLogs) (107380) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:12:26,696::__init__::232::IOProcess::(_processLogs) (107380) Queuing response >Thread-81::DEBUG::2014-09-25 17:12:26,696::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:12:26,697::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:26,698::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:26,698::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:26,699::__init__::232::IOProcess::(_processLogs) (107381) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:26,699::__init__::232::IOProcess::(_processLogs) (107381) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:26,699::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:26,700::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:26,700::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:26,700::__init__::232::IOProcess::(_processLogs) (107382) Got request for method 'glob' >Thread-81::DEBUG::2014-09-25 17:12:26,705::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:12:26,705::__init__::232::IOProcess::(_processLogs) (107382) Queuing response >Thread-81::INFO::2014-09-25 17:12:26,706::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:12:26,707::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:12:26,707::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:26,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:26,708::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:26,708::__init__::232::IOProcess::(_processLogs) (107383) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:26,713::__init__::232::IOProcess::(_processLogs) (107383) Queuing response >Thread-81::DEBUG::2014-09-25 17:12:26,714::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:12:26,730::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00326535 s, 114 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:26,731::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:26,732::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:26,732::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:26,732::__init__::232::IOProcess::(_processLogs) (107384) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:26,735::__init__::232::IOProcess::(_processLogs) (107384) Queuing response >Dummy-148::DEBUG::2014-09-25 17:12:27,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:27,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.219381 s, 4.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:28,726::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:28,728::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69807::DEBUG::2014-09-25 17:12:28,730::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:29,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:29,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157509 s, 65.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:31,095::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:31,095::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:31,095::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:31,095::__init__::232::IOProcess::(_processLogs) (107385) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:31,097::__init__::232::IOProcess::(_processLogs) (107385) Queuing response >Thread-79::DEBUG::2014-09-25 17:12:31,098::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:12:31,119::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00748687 s, 75.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:31,120::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:31,121::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:31,121::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:31,121::__init__::232::IOProcess::(_processLogs) (107386) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:31,127::__init__::232::IOProcess::(_processLogs) (107386) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:31,128::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:31,129::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:31,129::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:31,130::__init__::232::IOProcess::(_processLogs) (107387) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:31,130::__init__::232::IOProcess::(_processLogs) (107387) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:31,130::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:31,131::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:31,131::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:31,131::__init__::232::IOProcess::(_processLogs) (107388) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:31,131::__init__::232::IOProcess::(_processLogs) (107388) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:31,132::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:31,133::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:31,133::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:31,133::__init__::232::IOProcess::(_processLogs) (107389) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:31,133::__init__::232::IOProcess::(_processLogs) (107389) Queuing response >Dummy-148::DEBUG::2014-09-25 17:12:31,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:31,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133798 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:31,745::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:31,747::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69808::DEBUG::2014-09-25 17:12:31,750::task::595::Storage.TaskManager.Task::(_updateState) Task=`92b3e43a-4a6d-46de-9928-36868653549d`::moving from state init -> state preparing >Thread-69808::INFO::2014-09-25 17:12:31,750::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69808::INFO::2014-09-25 17:12:31,750::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00748687', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000533169', 'lastCheck': '9.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00326535', 'lastCheck': '5.0', 'valid': True}} >Thread-69808::DEBUG::2014-09-25 17:12:31,751::task::1191::Storage.TaskManager.Task::(prepare) Task=`92b3e43a-4a6d-46de-9928-36868653549d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00748687', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000533169', 'lastCheck': '9.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00326535', 'lastCheck': '5.0', 'valid': True}} >Thread-69808::DEBUG::2014-09-25 17:12:31,751::task::595::Storage.TaskManager.Task::(_updateState) Task=`92b3e43a-4a6d-46de-9928-36868653549d`::moving from state preparing -> state finished >Thread-69808::DEBUG::2014-09-25 17:12:31,751::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69808::DEBUG::2014-09-25 17:12:31,751::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69808::DEBUG::2014-09-25 17:12:31,752::task::993::Storage.TaskManager.Task::(_decref) Task=`92b3e43a-4a6d-46de-9928-36868653549d`::ref 0 aborting False >Thread-69808::DEBUG::2014-09-25 17:12:31,754::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:31,775::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:31,776::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69809::DEBUG::2014-09-25 17:12:31,778::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:12:32,395::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:32,396::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:32,396::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:32,398::__init__::232::IOProcess::(_processLogs) (107390) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:12:32,398::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:12:32,399::__init__::232::IOProcess::(_processLogs) (107390) Queuing response >Thread-80::DEBUG::2014-09-25 17:12:32,408::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00038159 s, 991 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:32,409::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:32,409::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:32,409::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:32,410::__init__::232::IOProcess::(_processLogs) (107391) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:32,410::__init__::232::IOProcess::(_processLogs) (107391) Queuing response >Dummy-148::DEBUG::2014-09-25 17:12:33,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:33,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194634 s, 52.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:34,827::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:34,829::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69810::DEBUG::2014-09-25 17:12:34,831::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:35,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:35,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0227441 s, 45.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:35,667::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:35,669::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69811::DEBUG::2014-09-25 17:12:35,669::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69811::DEBUG::2014-09-25 17:12:35,671::task::595::Storage.TaskManager.Task::(_updateState) Task=`93392880-e094-49ad-a23e-55419b37316b`::moving from state init -> state preparing >Thread-69811::INFO::2014-09-25 17:12:35,672::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69811::INFO::2014-09-25 17:12:35,672::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69811::DEBUG::2014-09-25 17:12:35,673::task::1191::Storage.TaskManager.Task::(prepare) Task=`93392880-e094-49ad-a23e-55419b37316b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69811::DEBUG::2014-09-25 17:12:35,673::task::595::Storage.TaskManager.Task::(_updateState) Task=`93392880-e094-49ad-a23e-55419b37316b`::moving from state preparing -> state finished >Thread-69811::DEBUG::2014-09-25 17:12:35,673::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69811::DEBUG::2014-09-25 17:12:35,673::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69811::DEBUG::2014-09-25 17:12:35,673::task::993::Storage.TaskManager.Task::(_decref) Task=`93392880-e094-49ad-a23e-55419b37316b`::ref 0 aborting False >Thread-69811::DEBUG::2014-09-25 17:12:35,674::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69811::DEBUG::2014-09-25 17:12:35,674::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:35,688::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69812::DEBUG::2014-09-25 17:12:35,689::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:12:35,689::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69812::DEBUG::2014-09-25 17:12:35,691::task::595::Storage.TaskManager.Task::(_updateState) Task=`70dc9990-a2e6-4f52-b287-1b7f5b663402`::moving from state init -> state preparing >Thread-69812::INFO::2014-09-25 17:12:35,692::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69812::DEBUG::2014-09-25 17:12:35,692::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`59e68482-3a3a-4bc3-93e8-86f6d8d464b9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69812::DEBUG::2014-09-25 17:12:35,693::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69812::DEBUG::2014-09-25 17:12:35,693::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69812::DEBUG::2014-09-25 17:12:35,693::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`59e68482-3a3a-4bc3-93e8-86f6d8d464b9`::Granted request >Thread-69812::DEBUG::2014-09-25 17:12:35,694::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`70dc9990-a2e6-4f52-b287-1b7f5b663402`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69812::DEBUG::2014-09-25 17:12:35,694::task::993::Storage.TaskManager.Task::(_decref) Task=`70dc9990-a2e6-4f52-b287-1b7f5b663402`::ref 1 aborting False >Thread-69812::INFO::2014-09-25 17:12:35,698::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69812::DEBUG::2014-09-25 17:12:35,699::task::1191::Storage.TaskManager.Task::(prepare) Task=`70dc9990-a2e6-4f52-b287-1b7f5b663402`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69812::DEBUG::2014-09-25 17:12:35,699::task::595::Storage.TaskManager.Task::(_updateState) Task=`70dc9990-a2e6-4f52-b287-1b7f5b663402`::moving from state preparing -> state finished >Thread-69812::DEBUG::2014-09-25 17:12:35,699::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69812::DEBUG::2014-09-25 17:12:35,699::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69812::DEBUG::2014-09-25 17:12:35,700::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69812::DEBUG::2014-09-25 17:12:35,700::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69812::DEBUG::2014-09-25 17:12:35,700::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69812::DEBUG::2014-09-25 17:12:35,700::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69812::DEBUG::2014-09-25 17:12:35,701::task::993::Storage.TaskManager.Task::(_decref) Task=`70dc9990-a2e6-4f52-b287-1b7f5b663402`::ref 0 aborting False >Thread-69812::DEBUG::2014-09-25 17:12:35,701::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69812::DEBUG::2014-09-25 17:12:35,702::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:12:36,737::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:36,738::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:36,738::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:36,738::__init__::232::IOProcess::(_processLogs) (107392) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:36,742::__init__::232::IOProcess::(_processLogs) (107392) Queuing response >Thread-81::DEBUG::2014-09-25 17:12:36,744::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:12:36,762::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00895147 s, 41.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:36,763::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:36,764::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:36,764::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:36,764::__init__::232::IOProcess::(_processLogs) (107393) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:36,765::__init__::232::IOProcess::(_processLogs) (107393) Queuing response >Dummy-148::DEBUG::2014-09-25 17:12:37,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:37,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.226225 s, 4.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:37,851::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:37,853::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69813::DEBUG::2014-09-25 17:12:37,855::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:39,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:39,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0227128 s, 45.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:40,872::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:40,873::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69814::DEBUG::2014-09-25 17:12:40,877::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:12:41,135::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:41,135::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:41,135::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:41,136::__init__::232::IOProcess::(_processLogs) (107394) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:41,136::__init__::232::IOProcess::(_processLogs) (107394) Queuing response >Thread-79::DEBUG::2014-09-25 17:12:41,137::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:12:41,145::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000516985 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:41,147::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:41,147::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:41,147::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:41,147::__init__::232::IOProcess::(_processLogs) (107395) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:41,148::__init__::232::IOProcess::(_processLogs) (107395) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:41,150::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:41,150::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:41,150::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:41,150::__init__::232::IOProcess::(_processLogs) (107396) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:41,151::__init__::232::IOProcess::(_processLogs) (107396) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:41,152::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:41,152::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:41,152::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:41,152::__init__::232::IOProcess::(_processLogs) (107397) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:41,153::__init__::232::IOProcess::(_processLogs) (107397) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:41,154::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:41,154::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:41,154::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:41,155::__init__::232::IOProcess::(_processLogs) (107398) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:41,155::__init__::232::IOProcess::(_processLogs) (107398) Queuing response >Dummy-148::DEBUG::2014-09-25 17:12:41,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:41,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170247 s, 60.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:42,413::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:42,413::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:42,413::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:42,413::__init__::232::IOProcess::(_processLogs) (107399) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:42,414::__init__::232::IOProcess::(_processLogs) (107399) Queuing response >Thread-80::DEBUG::2014-09-25 17:12:42,415::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:12:42,424::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000413677 s, 914 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:42,425::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:42,426::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:42,426::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:42,428::__init__::232::IOProcess::(_processLogs) (107400) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:42,428::__init__::232::IOProcess::(_processLogs) (107400) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:43,892::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:43,893::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69815::DEBUG::2014-09-25 17:12:43,896::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:43,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:43,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177339 s, 57.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:45,731::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69816::DEBUG::2014-09-25 17:12:45,733::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:12:45,733::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69816::DEBUG::2014-09-25 17:12:45,735::task::595::Storage.TaskManager.Task::(_updateState) Task=`f03f682a-d1fd-4d0e-a5e5-d07ea5acbeae`::moving from state init -> state preparing >Thread-69816::INFO::2014-09-25 17:12:45,735::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69816::INFO::2014-09-25 17:12:45,736::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69816::DEBUG::2014-09-25 17:12:45,736::task::1191::Storage.TaskManager.Task::(prepare) Task=`f03f682a-d1fd-4d0e-a5e5-d07ea5acbeae`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69816::DEBUG::2014-09-25 17:12:45,736::task::595::Storage.TaskManager.Task::(_updateState) Task=`f03f682a-d1fd-4d0e-a5e5-d07ea5acbeae`::moving from state preparing -> state finished >Thread-69816::DEBUG::2014-09-25 17:12:45,737::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69816::DEBUG::2014-09-25 17:12:45,737::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69816::DEBUG::2014-09-25 17:12:45,737::task::993::Storage.TaskManager.Task::(_decref) Task=`f03f682a-d1fd-4d0e-a5e5-d07ea5acbeae`::ref 0 aborting False >Thread-69816::DEBUG::2014-09-25 17:12:45,738::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69816::DEBUG::2014-09-25 17:12:45,738::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:45,750::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69817::DEBUG::2014-09-25 17:12:45,751::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:12:45,751::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69817::DEBUG::2014-09-25 17:12:45,754::task::595::Storage.TaskManager.Task::(_updateState) Task=`2a945d65-5bf5-4239-a8f8-d82e95780277`::moving from state init -> state preparing >Thread-69817::INFO::2014-09-25 17:12:45,754::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69817::DEBUG::2014-09-25 17:12:45,755::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ebe37dd0-1f4e-472e-aba9-9d52f93bbf61`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69817::DEBUG::2014-09-25 17:12:45,755::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69817::DEBUG::2014-09-25 17:12:45,755::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69817::DEBUG::2014-09-25 17:12:45,755::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ebe37dd0-1f4e-472e-aba9-9d52f93bbf61`::Granted request >Thread-69817::DEBUG::2014-09-25 17:12:45,756::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2a945d65-5bf5-4239-a8f8-d82e95780277`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69817::DEBUG::2014-09-25 17:12:45,756::task::993::Storage.TaskManager.Task::(_decref) Task=`2a945d65-5bf5-4239-a8f8-d82e95780277`::ref 1 aborting False >Thread-69817::INFO::2014-09-25 17:12:45,761::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69817::DEBUG::2014-09-25 17:12:45,761::task::1191::Storage.TaskManager.Task::(prepare) Task=`2a945d65-5bf5-4239-a8f8-d82e95780277`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69817::DEBUG::2014-09-25 17:12:45,762::task::595::Storage.TaskManager.Task::(_updateState) Task=`2a945d65-5bf5-4239-a8f8-d82e95780277`::moving from state preparing -> state finished >Thread-69817::DEBUG::2014-09-25 17:12:45,762::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69817::DEBUG::2014-09-25 17:12:45,762::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69817::DEBUG::2014-09-25 17:12:45,762::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69817::DEBUG::2014-09-25 17:12:45,763::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69817::DEBUG::2014-09-25 17:12:45,763::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69817::DEBUG::2014-09-25 17:12:45,763::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69817::DEBUG::2014-09-25 17:12:45,764::task::993::Storage.TaskManager.Task::(_decref) Task=`2a945d65-5bf5-4239-a8f8-d82e95780277`::ref 0 aborting False >Thread-69817::DEBUG::2014-09-25 17:12:45,764::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69817::DEBUG::2014-09-25 17:12:45,765::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:45,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:45,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147524 s, 69.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:46,768::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:46,770::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:46,771::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:46,772::__init__::232::IOProcess::(_processLogs) (107401) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:46,774::__init__::232::IOProcess::(_processLogs) (107401) Queuing response >Thread-81::DEBUG::2014-09-25 17:12:46,776::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:12:46,786::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000376314 s, 986 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:46,789::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:46,790::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:46,792::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:46,793::__init__::232::IOProcess::(_processLogs) (107402) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:46,795::__init__::232::IOProcess::(_processLogs) (107402) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:46,913::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:46,916::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69818::DEBUG::2014-09-25 17:12:46,924::task::595::Storage.TaskManager.Task::(_updateState) Task=`69811338-8138-448b-b5d2-0fb5965bbd04`::moving from state init -> state preparing >Thread-69818::INFO::2014-09-25 17:12:46,925::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69818::INFO::2014-09-25 17:12:46,927::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000516985', 'lastCheck': '5.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000413677', 'lastCheck': '4.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000376314', 'lastCheck': '0.1', 'valid': True}} >Thread-69818::DEBUG::2014-09-25 17:12:46,928::task::1191::Storage.TaskManager.Task::(prepare) Task=`69811338-8138-448b-b5d2-0fb5965bbd04`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000516985', 'lastCheck': '5.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000413677', 'lastCheck': '4.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000376314', 'lastCheck': '0.1', 'valid': True}} >Thread-69818::DEBUG::2014-09-25 17:12:46,929::task::595::Storage.TaskManager.Task::(_updateState) Task=`69811338-8138-448b-b5d2-0fb5965bbd04`::moving from state preparing -> state finished >Thread-69818::DEBUG::2014-09-25 17:12:46,930::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69818::DEBUG::2014-09-25 17:12:46,931::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69818::DEBUG::2014-09-25 17:12:46,932::task::993::Storage.TaskManager.Task::(_decref) Task=`69811338-8138-448b-b5d2-0fb5965bbd04`::ref 0 aborting False >Thread-69818::DEBUG::2014-09-25 17:12:46,939::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:46,964::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:46,967::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69819::DEBUG::2014-09-25 17:12:46,972::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:48,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:48,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0190383 s, 53.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:50,024::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:50,026::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69820::DEBUG::2014-09-25 17:12:50,028::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:50,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:50,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130049 s, 78.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:51,157::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:51,158::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-79::DEBUG::2014-09-25 17:12:51,159::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:12:51,159::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:51,168::__init__::232::IOProcess::(_processLogs) (107403) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:51,168::__init__::232::IOProcess::(_processLogs) (107403) Queuing response >Thread-79::DEBUG::2014-09-25 17:12:51,169::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000716172 s, 785 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:51,170::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:51,170::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:51,170::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:51,171::__init__::232::IOProcess::(_processLogs) (107404) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:51,172::__init__::232::IOProcess::(_processLogs) (107404) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:51,173::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:51,173::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:51,174::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:51,174::__init__::232::IOProcess::(_processLogs) (107405) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:51,174::__init__::232::IOProcess::(_processLogs) (107405) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:51,175::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:51,175::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:51,176::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:51,176::__init__::232::IOProcess::(_processLogs) (107406) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:51,176::__init__::232::IOProcess::(_processLogs) (107406) Queuing response >Thread-29::DEBUG::2014-09-25 17:12:51,177::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:51,177::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:51,178::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:51,179::__init__::232::IOProcess::(_processLogs) (107407) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:12:51,179::__init__::232::IOProcess::(_processLogs) (107407) Queuing response >Dummy-148::DEBUG::2014-09-25 17:12:52,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:52,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152808 s, 67.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:52,428::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:52,429::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:52,429::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:52,429::__init__::232::IOProcess::(_processLogs) (107408) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:52,430::__init__::232::IOProcess::(_processLogs) (107408) Queuing response >Thread-80::DEBUG::2014-09-25 17:12:52,431::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:12:52,439::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000377961 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:52,441::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:52,441::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:52,441::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:52,441::__init__::232::IOProcess::(_processLogs) (107409) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:52,442::__init__::232::IOProcess::(_processLogs) (107409) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:53,042::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:53,043::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69821::DEBUG::2014-09-25 17:12:53,045::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:54,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:54,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210749 s, 48.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:55,794::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:55,796::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69822::DEBUG::2014-09-25 17:12:55,796::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69822::DEBUG::2014-09-25 17:12:55,798::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a811c29-c732-4a14-b961-440fe49ea8ff`::moving from state init -> state preparing >Thread-69822::INFO::2014-09-25 17:12:55,798::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69822::INFO::2014-09-25 17:12:55,799::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69822::DEBUG::2014-09-25 17:12:55,799::task::1191::Storage.TaskManager.Task::(prepare) Task=`4a811c29-c732-4a14-b961-440fe49ea8ff`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69822::DEBUG::2014-09-25 17:12:55,799::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a811c29-c732-4a14-b961-440fe49ea8ff`::moving from state preparing -> state finished >Thread-69822::DEBUG::2014-09-25 17:12:55,800::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69822::DEBUG::2014-09-25 17:12:55,800::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69822::DEBUG::2014-09-25 17:12:55,800::task::993::Storage.TaskManager.Task::(_decref) Task=`4a811c29-c732-4a14-b961-440fe49ea8ff`::ref 0 aborting False >Thread-69822::DEBUG::2014-09-25 17:12:55,800::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69822::DEBUG::2014-09-25 17:12:55,801::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:55,812::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69823::DEBUG::2014-09-25 17:12:55,813::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:12:55,813::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69823::DEBUG::2014-09-25 17:12:55,815::task::595::Storage.TaskManager.Task::(_updateState) Task=`bd73e75f-0dc9-425f-893d-d73bb98182ac`::moving from state init -> state preparing >Thread-69823::INFO::2014-09-25 17:12:55,816::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69823::DEBUG::2014-09-25 17:12:55,816::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b14646d6-c6a5-483a-86b0-2c73cd6c2751`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69823::DEBUG::2014-09-25 17:12:55,817::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69823::DEBUG::2014-09-25 17:12:55,817::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69823::DEBUG::2014-09-25 17:12:55,817::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b14646d6-c6a5-483a-86b0-2c73cd6c2751`::Granted request >Thread-69823::DEBUG::2014-09-25 17:12:55,818::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`bd73e75f-0dc9-425f-893d-d73bb98182ac`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69823::DEBUG::2014-09-25 17:12:55,818::task::993::Storage.TaskManager.Task::(_decref) Task=`bd73e75f-0dc9-425f-893d-d73bb98182ac`::ref 1 aborting False >Thread-69823::INFO::2014-09-25 17:12:55,823::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69823::DEBUG::2014-09-25 17:12:55,823::task::1191::Storage.TaskManager.Task::(prepare) Task=`bd73e75f-0dc9-425f-893d-d73bb98182ac`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69823::DEBUG::2014-09-25 17:12:55,823::task::595::Storage.TaskManager.Task::(_updateState) Task=`bd73e75f-0dc9-425f-893d-d73bb98182ac`::moving from state preparing -> state finished >Thread-69823::DEBUG::2014-09-25 17:12:55,823::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69823::DEBUG::2014-09-25 17:12:55,824::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69823::DEBUG::2014-09-25 17:12:55,824::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69823::DEBUG::2014-09-25 17:12:55,824::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69823::DEBUG::2014-09-25 17:12:55,825::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69823::DEBUG::2014-09-25 17:12:55,825::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69823::DEBUG::2014-09-25 17:12:55,825::task::993::Storage.TaskManager.Task::(_decref) Task=`bd73e75f-0dc9-425f-893d-d73bb98182ac`::ref 0 aborting False >Thread-69823::DEBUG::2014-09-25 17:12:55,826::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69823::DEBUG::2014-09-25 17:12:55,826::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:56,059::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:56,062::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69824::DEBUG::2014-09-25 17:12:56,064::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:12:56,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:56,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157041 s, 65.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:56,798::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:56,798::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:56,799::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:56,799::__init__::232::IOProcess::(_processLogs) (107410) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:56,799::__init__::232::IOProcess::(_processLogs) (107410) Queuing response >Thread-81::DEBUG::2014-09-25 17:12:56,801::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:12:56,809::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000342404 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:12:56,811::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:12:56,811::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:12:56,811::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:12:56,811::__init__::232::IOProcess::(_processLogs) (107411) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:12:56,812::__init__::232::IOProcess::(_processLogs) (107411) Queuing response >Dummy-148::DEBUG::2014-09-25 17:12:58,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:12:58,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162938 s, 62.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:12:59,079::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:12:59,080::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69825::DEBUG::2014-09-25 17:12:59,082::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:00,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:00,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132735 s, 77.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:01,179::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:01,180::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:01,180::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:01,180::__init__::232::IOProcess::(_processLogs) (107412) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:01,183::__init__::232::IOProcess::(_processLogs) (107412) Queuing response >Thread-79::DEBUG::2014-09-25 17:13:01,184::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:13:01,217::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0121291 s, 46.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:01,218::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:01,219::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:01,219::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:01,219::__init__::232::IOProcess::(_processLogs) (107413) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:01,227::__init__::232::IOProcess::(_processLogs) (107413) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:01,228::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:01,228::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:01,229::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:01,229::__init__::232::IOProcess::(_processLogs) (107414) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:01,229::__init__::232::IOProcess::(_processLogs) (107414) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:01,230::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:01,230::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:01,231::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:01,231::__init__::232::IOProcess::(_processLogs) (107415) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:01,232::__init__::232::IOProcess::(_processLogs) (107415) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:01,232::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:01,233::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:01,234::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:01,234::__init__::232::IOProcess::(_processLogs) (107416) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:01,234::__init__::232::IOProcess::(_processLogs) (107416) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:02,096::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:02,098::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69826::DEBUG::2014-09-25 17:13:02,100::task::595::Storage.TaskManager.Task::(_updateState) Task=`d39e9beb-abca-40ee-8e37-6e703222c626`::moving from state init -> state preparing >Thread-69826::INFO::2014-09-25 17:13:02,101::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69826::INFO::2014-09-25 17:13:02,101::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0121291', 'lastCheck': '0.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000377961', 'lastCheck': '9.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000342404', 'lastCheck': '5.3', 'valid': True}} >Thread-69826::DEBUG::2014-09-25 17:13:02,101::task::1191::Storage.TaskManager.Task::(prepare) Task=`d39e9beb-abca-40ee-8e37-6e703222c626`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0121291', 'lastCheck': '0.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000377961', 'lastCheck': '9.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000342404', 'lastCheck': '5.3', 'valid': True}} >Thread-69826::DEBUG::2014-09-25 17:13:02,102::task::595::Storage.TaskManager.Task::(_updateState) Task=`d39e9beb-abca-40ee-8e37-6e703222c626`::moving from state preparing -> state finished >Thread-69826::DEBUG::2014-09-25 17:13:02,102::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69826::DEBUG::2014-09-25 17:13:02,102::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69826::DEBUG::2014-09-25 17:13:02,102::task::993::Storage.TaskManager.Task::(_decref) Task=`d39e9beb-abca-40ee-8e37-6e703222c626`::ref 0 aborting False >Thread-69826::DEBUG::2014-09-25 17:13:02,105::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:02,118::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:02,119::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69827::DEBUG::2014-09-25 17:13:02,121::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:02,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:02,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021136 s, 48.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:02,444::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:02,444::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:02,445::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:13:02,446::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:13:02,446::__init__::232::IOProcess::(_processLogs) (107417) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:02,454::__init__::232::IOProcess::(_processLogs) (107417) Queuing response >Thread-80::DEBUG::2014-09-25 17:13:02,455::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000367274 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:02,457::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:02,457::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:02,457::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:02,457::__init__::232::IOProcess::(_processLogs) (107418) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:02,458::__init__::232::IOProcess::(_processLogs) (107418) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:04,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:04,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.218824 s, 4.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:05,159::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:05,161::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69828::DEBUG::2014-09-25 17:13:05,163::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:05,856::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69829::DEBUG::2014-09-25 17:13:05,857::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:13:05,857::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69829::DEBUG::2014-09-25 17:13:05,859::task::595::Storage.TaskManager.Task::(_updateState) Task=`67aceb1b-1cfa-4b40-be46-a64717676340`::moving from state init -> state preparing >Thread-69829::INFO::2014-09-25 17:13:05,859::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69829::INFO::2014-09-25 17:13:05,860::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69829::DEBUG::2014-09-25 17:13:05,860::task::1191::Storage.TaskManager.Task::(prepare) Task=`67aceb1b-1cfa-4b40-be46-a64717676340`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69829::DEBUG::2014-09-25 17:13:05,860::task::595::Storage.TaskManager.Task::(_updateState) Task=`67aceb1b-1cfa-4b40-be46-a64717676340`::moving from state preparing -> state finished >Thread-69829::DEBUG::2014-09-25 17:13:05,860::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69829::DEBUG::2014-09-25 17:13:05,861::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69829::DEBUG::2014-09-25 17:13:05,861::task::993::Storage.TaskManager.Task::(_decref) Task=`67aceb1b-1cfa-4b40-be46-a64717676340`::ref 0 aborting False >Thread-69829::DEBUG::2014-09-25 17:13:05,861::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69829::DEBUG::2014-09-25 17:13:05,862::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:05,876::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69830::DEBUG::2014-09-25 17:13:05,877::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:13:05,877::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69830::DEBUG::2014-09-25 17:13:05,879::task::595::Storage.TaskManager.Task::(_updateState) Task=`c7fccea9-f578-4558-8552-d7da1eff20fb`::moving from state init -> state preparing >Thread-69830::INFO::2014-09-25 17:13:05,879::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69830::DEBUG::2014-09-25 17:13:05,880::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a6d4d82f-69e9-4592-8328-95e274a680b1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69830::DEBUG::2014-09-25 17:13:05,880::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69830::DEBUG::2014-09-25 17:13:05,881::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69830::DEBUG::2014-09-25 17:13:05,881::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a6d4d82f-69e9-4592-8328-95e274a680b1`::Granted request >Thread-69830::DEBUG::2014-09-25 17:13:05,881::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c7fccea9-f578-4558-8552-d7da1eff20fb`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69830::DEBUG::2014-09-25 17:13:05,882::task::993::Storage.TaskManager.Task::(_decref) Task=`c7fccea9-f578-4558-8552-d7da1eff20fb`::ref 1 aborting False >Thread-69830::INFO::2014-09-25 17:13:05,886::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69830::DEBUG::2014-09-25 17:13:05,886::task::1191::Storage.TaskManager.Task::(prepare) Task=`c7fccea9-f578-4558-8552-d7da1eff20fb`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69830::DEBUG::2014-09-25 17:13:05,887::task::595::Storage.TaskManager.Task::(_updateState) Task=`c7fccea9-f578-4558-8552-d7da1eff20fb`::moving from state preparing -> state finished >Thread-69830::DEBUG::2014-09-25 17:13:05,887::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69830::DEBUG::2014-09-25 17:13:05,887::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69830::DEBUG::2014-09-25 17:13:05,887::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69830::DEBUG::2014-09-25 17:13:05,888::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69830::DEBUG::2014-09-25 17:13:05,888::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69830::DEBUG::2014-09-25 17:13:05,888::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69830::DEBUG::2014-09-25 17:13:05,888::task::993::Storage.TaskManager.Task::(_decref) Task=`c7fccea9-f578-4558-8552-d7da1eff20fb`::ref 0 aborting False >Thread-69830::DEBUG::2014-09-25 17:13:05,889::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69830::DEBUG::2014-09-25 17:13:05,889::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:06,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:06,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203805 s, 50.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:06,814::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:06,815::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:06,815::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:06,815::__init__::232::IOProcess::(_processLogs) (107419) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:06,816::__init__::232::IOProcess::(_processLogs) (107419) Queuing response >Thread-81::DEBUG::2014-09-25 17:13:06,817::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:13:06,825::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000356908 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:06,826::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:06,827::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:06,827::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:06,827::__init__::232::IOProcess::(_processLogs) (107420) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:06,828::__init__::232::IOProcess::(_processLogs) (107420) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:08,177::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:08,179::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69831::DEBUG::2014-09-25 17:13:08,181::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:08,721::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:08,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0375587 s, 27.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:13:10,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:10,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0140748 s, 72.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:11,198::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:11,199::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69832::DEBUG::2014-09-25 17:13:11,201::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:13:11,234::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:11,235::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:11,235::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:11,235::__init__::232::IOProcess::(_processLogs) (107421) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:11,238::__init__::232::IOProcess::(_processLogs) (107421) Queuing response >Thread-79::DEBUG::2014-09-25 17:13:11,239::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:13:11,251::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000885337 s, 635 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:11,252::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:11,252::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:11,252::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:11,253::__init__::232::IOProcess::(_processLogs) (107422) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:11,254::__init__::232::IOProcess::(_processLogs) (107422) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:11,255::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:11,256::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:11,256::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:11,257::__init__::232::IOProcess::(_processLogs) (107423) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:11,257::__init__::232::IOProcess::(_processLogs) (107423) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:11,257::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:11,258::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:11,258::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:11,259::__init__::232::IOProcess::(_processLogs) (107424) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:11,259::__init__::232::IOProcess::(_processLogs) (107424) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:11,259::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:11,260::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:11,260::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:11,261::__init__::232::IOProcess::(_processLogs) (107425) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:11,261::__init__::232::IOProcess::(_processLogs) (107425) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:12,460::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:12,460::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:12,461::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:12,461::__init__::232::IOProcess::(_processLogs) (107426) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:12,461::__init__::232::IOProcess::(_processLogs) (107426) Queuing response >Thread-80::DEBUG::2014-09-25 17:13:12,462::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:13:12,472::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000559785 s, 675 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:12,473::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:12,473::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:12,473::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:12,474::__init__::232::IOProcess::(_processLogs) (107427) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:12,474::__init__::232::IOProcess::(_processLogs) (107427) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:12,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:12,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0198931 s, 51.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:14,216::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:14,221::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69833::DEBUG::2014-09-25 17:13:14,225::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:14,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:14,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129559 s, 79.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:15,918::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69834::DEBUG::2014-09-25 17:13:15,919::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:13:15,919::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69834::DEBUG::2014-09-25 17:13:15,922::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b0ec232-f647-4f4f-a6de-da1ac455f182`::moving from state init -> state preparing >Thread-69834::INFO::2014-09-25 17:13:15,922::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69834::INFO::2014-09-25 17:13:15,923::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69834::DEBUG::2014-09-25 17:13:15,923::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b0ec232-f647-4f4f-a6de-da1ac455f182`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69834::DEBUG::2014-09-25 17:13:15,923::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b0ec232-f647-4f4f-a6de-da1ac455f182`::moving from state preparing -> state finished >Thread-69834::DEBUG::2014-09-25 17:13:15,924::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69834::DEBUG::2014-09-25 17:13:15,924::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69834::DEBUG::2014-09-25 17:13:15,924::task::993::Storage.TaskManager.Task::(_decref) Task=`1b0ec232-f647-4f4f-a6de-da1ac455f182`::ref 0 aborting False >Thread-69834::DEBUG::2014-09-25 17:13:15,924::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69834::DEBUG::2014-09-25 17:13:15,925::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:15,935::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69835::DEBUG::2014-09-25 17:13:15,937::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:13:15,937::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69835::DEBUG::2014-09-25 17:13:15,939::task::595::Storage.TaskManager.Task::(_updateState) Task=`81046a08-e5ec-44bd-9644-7eedd1d666f7`::moving from state init -> state preparing >Thread-69835::INFO::2014-09-25 17:13:15,939::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69835::DEBUG::2014-09-25 17:13:15,940::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`819a0f57-9fcc-48fe-b256-24ecc771b36b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69835::DEBUG::2014-09-25 17:13:15,940::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69835::DEBUG::2014-09-25 17:13:15,941::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69835::DEBUG::2014-09-25 17:13:15,941::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`819a0f57-9fcc-48fe-b256-24ecc771b36b`::Granted request >Thread-69835::DEBUG::2014-09-25 17:13:15,942::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`81046a08-e5ec-44bd-9644-7eedd1d666f7`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69835::DEBUG::2014-09-25 17:13:15,942::task::993::Storage.TaskManager.Task::(_decref) Task=`81046a08-e5ec-44bd-9644-7eedd1d666f7`::ref 1 aborting False >Thread-69835::INFO::2014-09-25 17:13:15,946::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69835::DEBUG::2014-09-25 17:13:15,947::task::1191::Storage.TaskManager.Task::(prepare) Task=`81046a08-e5ec-44bd-9644-7eedd1d666f7`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69835::DEBUG::2014-09-25 17:13:15,947::task::595::Storage.TaskManager.Task::(_updateState) Task=`81046a08-e5ec-44bd-9644-7eedd1d666f7`::moving from state preparing -> state finished >Thread-69835::DEBUG::2014-09-25 17:13:15,947::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69835::DEBUG::2014-09-25 17:13:15,947::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69835::DEBUG::2014-09-25 17:13:15,948::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69835::DEBUG::2014-09-25 17:13:15,948::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69835::DEBUG::2014-09-25 17:13:15,948::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69835::DEBUG::2014-09-25 17:13:15,949::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69835::DEBUG::2014-09-25 17:13:15,949::task::993::Storage.TaskManager.Task::(_decref) Task=`81046a08-e5ec-44bd-9644-7eedd1d666f7`::ref 0 aborting False >Thread-69835::DEBUG::2014-09-25 17:13:15,949::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69835::DEBUG::2014-09-25 17:13:15,950::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:13:16,831::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:16,832::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:16,833::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:16,835::__init__::232::IOProcess::(_processLogs) (107428) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:16,837::__init__::232::IOProcess::(_processLogs) (107428) Queuing response >Thread-81::DEBUG::2014-09-25 17:13:16,838::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:13:16,858::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00382388 s, 97.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:16,861::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:16,862::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:16,863::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:16,865::__init__::232::IOProcess::(_processLogs) (107429) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:16,867::__init__::232::IOProcess::(_processLogs) (107429) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:16,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:16,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0259147 s, 39.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:17,239::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:17,241::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69836::DEBUG::2014-09-25 17:13:17,243::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a501efe-7a6f-4805-b5c4-e463bd539955`::moving from state init -> state preparing >Thread-69836::INFO::2014-09-25 17:13:17,244::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69836::INFO::2014-09-25 17:13:17,244::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000885337', 'lastCheck': '6.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000559785', 'lastCheck': '4.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00382388', 'lastCheck': '0.4', 'valid': True}} >Thread-69836::DEBUG::2014-09-25 17:13:17,244::task::1191::Storage.TaskManager.Task::(prepare) Task=`9a501efe-7a6f-4805-b5c4-e463bd539955`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000885337', 'lastCheck': '6.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000559785', 'lastCheck': '4.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00382388', 'lastCheck': '0.4', 'valid': True}} >Thread-69836::DEBUG::2014-09-25 17:13:17,245::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a501efe-7a6f-4805-b5c4-e463bd539955`::moving from state preparing -> state finished >Thread-69836::DEBUG::2014-09-25 17:13:17,245::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69836::DEBUG::2014-09-25 17:13:17,245::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69836::DEBUG::2014-09-25 17:13:17,245::task::993::Storage.TaskManager.Task::(_decref) Task=`9a501efe-7a6f-4805-b5c4-e463bd539955`::ref 0 aborting False >Thread-69836::DEBUG::2014-09-25 17:13:17,247::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:17,262::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:17,263::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69837::DEBUG::2014-09-25 17:13:17,265::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:19,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:19,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021462 s, 47.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:20,307::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:20,308::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69838::DEBUG::2014-09-25 17:13:20,310::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:21,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:21,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013195 s, 77.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:21,262::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:21,262::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:21,263::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:21,263::__init__::232::IOProcess::(_processLogs) (107430) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:21,264::__init__::232::IOProcess::(_processLogs) (107430) Queuing response >Thread-79::DEBUG::2014-09-25 17:13:21,265::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:13:21,275::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00104121 s, 540 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:21,276::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:21,276::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:21,276::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:21,277::__init__::232::IOProcess::(_processLogs) (107431) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:21,278::__init__::232::IOProcess::(_processLogs) (107431) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:21,279::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:21,279::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:21,280::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:21,281::__init__::232::IOProcess::(_processLogs) (107432) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:21,281::__init__::232::IOProcess::(_processLogs) (107432) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:21,281::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:21,282::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:21,282::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:21,283::__init__::232::IOProcess::(_processLogs) (107433) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:21,283::__init__::232::IOProcess::(_processLogs) (107433) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:21,283::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:21,284::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:21,284::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:21,285::__init__::232::IOProcess::(_processLogs) (107434) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:21,285::__init__::232::IOProcess::(_processLogs) (107434) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:22,476::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:22,477::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:22,477::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:22,478::__init__::232::IOProcess::(_processLogs) (107435) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:13:22,478::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:13:22,480::__init__::232::IOProcess::(_processLogs) (107435) Queuing response >Thread-80::DEBUG::2014-09-25 17:13:22,488::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000449712 s, 841 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:22,489::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:22,489::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:22,490::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:22,490::__init__::232::IOProcess::(_processLogs) (107436) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:22,491::__init__::232::IOProcess::(_processLogs) (107436) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:23,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:23,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218883 s, 46.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:23,324::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:23,326::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69839::DEBUG::2014-09-25 17:13:23,328::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:25,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:25,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158445 s, 64.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:25,977::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69840::DEBUG::2014-09-25 17:13:25,979::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:13:25,979::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69840::DEBUG::2014-09-25 17:13:25,981::task::595::Storage.TaskManager.Task::(_updateState) Task=`58f6a58c-25b1-4910-8ffb-75ee48868432`::moving from state init -> state preparing >Thread-69840::INFO::2014-09-25 17:13:25,982::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69840::INFO::2014-09-25 17:13:25,982::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69840::DEBUG::2014-09-25 17:13:25,982::task::1191::Storage.TaskManager.Task::(prepare) Task=`58f6a58c-25b1-4910-8ffb-75ee48868432`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69840::DEBUG::2014-09-25 17:13:25,983::task::595::Storage.TaskManager.Task::(_updateState) Task=`58f6a58c-25b1-4910-8ffb-75ee48868432`::moving from state preparing -> state finished >Thread-69840::DEBUG::2014-09-25 17:13:25,983::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69840::DEBUG::2014-09-25 17:13:25,983::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69840::DEBUG::2014-09-25 17:13:25,983::task::993::Storage.TaskManager.Task::(_decref) Task=`58f6a58c-25b1-4910-8ffb-75ee48868432`::ref 0 aborting False >Thread-69840::DEBUG::2014-09-25 17:13:25,984::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69840::DEBUG::2014-09-25 17:13:25,984::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:25,999::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:26,000::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69841::DEBUG::2014-09-25 17:13:26,000::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69841::DEBUG::2014-09-25 17:13:26,002::task::595::Storage.TaskManager.Task::(_updateState) Task=`a19ffb85-ec21-4d55-90fd-b4913a607e97`::moving from state init -> state preparing >Thread-69841::INFO::2014-09-25 17:13:26,002::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69841::DEBUG::2014-09-25 17:13:26,003::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`20a2be3b-d9be-47c8-b33f-e1a2c2327c52`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69841::DEBUG::2014-09-25 17:13:26,003::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69841::DEBUG::2014-09-25 17:13:26,004::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69841::DEBUG::2014-09-25 17:13:26,004::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`20a2be3b-d9be-47c8-b33f-e1a2c2327c52`::Granted request >Thread-69841::DEBUG::2014-09-25 17:13:26,005::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a19ffb85-ec21-4d55-90fd-b4913a607e97`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69841::DEBUG::2014-09-25 17:13:26,006::task::993::Storage.TaskManager.Task::(_decref) Task=`a19ffb85-ec21-4d55-90fd-b4913a607e97`::ref 1 aborting False >Thread-69841::INFO::2014-09-25 17:13:26,010::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69841::DEBUG::2014-09-25 17:13:26,010::task::1191::Storage.TaskManager.Task::(prepare) Task=`a19ffb85-ec21-4d55-90fd-b4913a607e97`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69841::DEBUG::2014-09-25 17:13:26,010::task::595::Storage.TaskManager.Task::(_updateState) Task=`a19ffb85-ec21-4d55-90fd-b4913a607e97`::moving from state preparing -> state finished >Thread-69841::DEBUG::2014-09-25 17:13:26,011::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69841::DEBUG::2014-09-25 17:13:26,011::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69841::DEBUG::2014-09-25 17:13:26,011::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69841::DEBUG::2014-09-25 17:13:26,012::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69841::DEBUG::2014-09-25 17:13:26,012::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69841::DEBUG::2014-09-25 17:13:26,012::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69841::DEBUG::2014-09-25 17:13:26,012::task::993::Storage.TaskManager.Task::(_decref) Task=`a19ffb85-ec21-4d55-90fd-b4913a607e97`::ref 0 aborting False >Thread-69841::DEBUG::2014-09-25 17:13:26,013::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69841::DEBUG::2014-09-25 17:13:26,013::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:26,341::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:26,343::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69842::DEBUG::2014-09-25 17:13:26,344::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:13:26,869::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:26,870::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:26,870::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:26,870::__init__::232::IOProcess::(_processLogs) (107437) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:26,875::__init__::232::IOProcess::(_processLogs) (107437) Queuing response >Thread-81::DEBUG::2014-09-25 17:13:26,876::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:13:26,892::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00191844 s, 193 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:26,893::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:26,893::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:26,893::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:26,894::__init__::232::IOProcess::(_processLogs) (107438) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:26,897::__init__::232::IOProcess::(_processLogs) (107438) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:27,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:27,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209439 s, 48.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:13:29,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:29,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0296427 s, 34.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:29,357::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:29,359::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69843::DEBUG::2014-09-25 17:13:29,361::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:13:31,286::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:31,286::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:31,286::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:31,286::__init__::232::IOProcess::(_processLogs) (107439) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:31,289::__init__::232::IOProcess::(_processLogs) (107439) Queuing response >Thread-79::DEBUG::2014-09-25 17:13:31,290::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:13:31,307::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00328095 s, 171 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:31,309::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:31,309::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:31,309::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:31,310::__init__::232::IOProcess::(_processLogs) (107440) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:31,313::__init__::232::IOProcess::(_processLogs) (107440) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:31,314::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:31,315::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:31,315::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:31,315::__init__::232::IOProcess::(_processLogs) (107441) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:31,315::__init__::232::IOProcess::(_processLogs) (107441) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:31,317::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:31,317::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:31,318::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:31,318::__init__::232::IOProcess::(_processLogs) (107442) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:31,318::__init__::232::IOProcess::(_processLogs) (107442) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:31,319::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:31,320::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:31,320::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:31,320::__init__::232::IOProcess::(_processLogs) (107443) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:31,320::__init__::232::IOProcess::(_processLogs) (107443) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:31,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:31,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203105 s, 50.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:32,373::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:32,375::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69844::DEBUG::2014-09-25 17:13:32,377::task::595::Storage.TaskManager.Task::(_updateState) Task=`b30a0295-b7d8-40ed-af19-371444b4ce00`::moving from state init -> state preparing >Thread-69844::INFO::2014-09-25 17:13:32,378::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69844::INFO::2014-09-25 17:13:32,378::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00328095', 'lastCheck': '1.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449712', 'lastCheck': '9.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00191844', 'lastCheck': '5.5', 'valid': True}} >Thread-69844::DEBUG::2014-09-25 17:13:32,378::task::1191::Storage.TaskManager.Task::(prepare) Task=`b30a0295-b7d8-40ed-af19-371444b4ce00`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00328095', 'lastCheck': '1.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449712', 'lastCheck': '9.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00191844', 'lastCheck': '5.5', 'valid': True}} >Thread-69844::DEBUG::2014-09-25 17:13:32,379::task::595::Storage.TaskManager.Task::(_updateState) Task=`b30a0295-b7d8-40ed-af19-371444b4ce00`::moving from state preparing -> state finished >Thread-69844::DEBUG::2014-09-25 17:13:32,379::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69844::DEBUG::2014-09-25 17:13:32,379::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69844::DEBUG::2014-09-25 17:13:32,379::task::993::Storage.TaskManager.Task::(_decref) Task=`b30a0295-b7d8-40ed-af19-371444b4ce00`::ref 0 aborting False >Thread-69844::DEBUG::2014-09-25 17:13:32,382::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:32,404::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:32,406::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69845::DEBUG::2014-09-25 17:13:32,407::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:13:32,493::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:32,493::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:32,494::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:32,494::__init__::232::IOProcess::(_processLogs) (107444) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:32,494::__init__::232::IOProcess::(_processLogs) (107444) Queuing response >Thread-80::DEBUG::2014-09-25 17:13:32,495::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:13:32,504::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000497214 s, 760 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:32,505::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:32,506::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:32,506::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:32,506::__init__::232::IOProcess::(_processLogs) (107445) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:32,507::__init__::232::IOProcess::(_processLogs) (107445) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:33,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:33,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188299 s, 54.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:13:35,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:35,472::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:35,475::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69846::DEBUG::2014-09-25 17:13:35,481::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:35,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0284904 s, 35.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:36,047::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:36,048::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69847::DEBUG::2014-09-25 17:13:36,048::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69847::DEBUG::2014-09-25 17:13:36,050::task::595::Storage.TaskManager.Task::(_updateState) Task=`63e42886-a676-4c40-bdb1-24561a154d17`::moving from state init -> state preparing >Thread-69847::INFO::2014-09-25 17:13:36,051::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69847::INFO::2014-09-25 17:13:36,051::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69847::DEBUG::2014-09-25 17:13:36,051::task::1191::Storage.TaskManager.Task::(prepare) Task=`63e42886-a676-4c40-bdb1-24561a154d17`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69847::DEBUG::2014-09-25 17:13:36,052::task::595::Storage.TaskManager.Task::(_updateState) Task=`63e42886-a676-4c40-bdb1-24561a154d17`::moving from state preparing -> state finished >Thread-69847::DEBUG::2014-09-25 17:13:36,052::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69847::DEBUG::2014-09-25 17:13:36,052::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69847::DEBUG::2014-09-25 17:13:36,052::task::993::Storage.TaskManager.Task::(_decref) Task=`63e42886-a676-4c40-bdb1-24561a154d17`::ref 0 aborting False >Thread-69847::DEBUG::2014-09-25 17:13:36,053::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69847::DEBUG::2014-09-25 17:13:36,053::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:36,065::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:36,067::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69848::DEBUG::2014-09-25 17:13:36,067::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69848::DEBUG::2014-09-25 17:13:36,068::task::595::Storage.TaskManager.Task::(_updateState) Task=`24d8e4e0-c9fc-4119-958b-d344629136c4`::moving from state init -> state preparing >Thread-69848::INFO::2014-09-25 17:13:36,069::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69848::DEBUG::2014-09-25 17:13:36,069::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2ef061a6-1211-4194-a095-6e33e86ab1de`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69848::DEBUG::2014-09-25 17:13:36,070::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69848::DEBUG::2014-09-25 17:13:36,070::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69848::DEBUG::2014-09-25 17:13:36,070::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2ef061a6-1211-4194-a095-6e33e86ab1de`::Granted request >Thread-69848::DEBUG::2014-09-25 17:13:36,071::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`24d8e4e0-c9fc-4119-958b-d344629136c4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69848::DEBUG::2014-09-25 17:13:36,071::task::993::Storage.TaskManager.Task::(_decref) Task=`24d8e4e0-c9fc-4119-958b-d344629136c4`::ref 1 aborting False >Thread-69848::INFO::2014-09-25 17:13:36,075::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69848::DEBUG::2014-09-25 17:13:36,076::task::1191::Storage.TaskManager.Task::(prepare) Task=`24d8e4e0-c9fc-4119-958b-d344629136c4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69848::DEBUG::2014-09-25 17:13:36,076::task::595::Storage.TaskManager.Task::(_updateState) Task=`24d8e4e0-c9fc-4119-958b-d344629136c4`::moving from state preparing -> state finished >Thread-69848::DEBUG::2014-09-25 17:13:36,076::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69848::DEBUG::2014-09-25 17:13:36,076::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69848::DEBUG::2014-09-25 17:13:36,077::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69848::DEBUG::2014-09-25 17:13:36,077::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69848::DEBUG::2014-09-25 17:13:36,077::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69848::DEBUG::2014-09-25 17:13:36,078::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69848::DEBUG::2014-09-25 17:13:36,078::task::993::Storage.TaskManager.Task::(_decref) Task=`24d8e4e0-c9fc-4119-958b-d344629136c4`::ref 0 aborting False >Thread-69848::DEBUG::2014-09-25 17:13:36,078::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69848::DEBUG::2014-09-25 17:13:36,079::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:13:36,899::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:36,899::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:36,900::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:36,900::__init__::232::IOProcess::(_processLogs) (107446) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:36,900::__init__::232::IOProcess::(_processLogs) (107446) Queuing response >Thread-81::DEBUG::2014-09-25 17:13:36,902::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:13:36,910::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000355542 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:36,911::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:36,911::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:36,912::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:36,912::__init__::232::IOProcess::(_processLogs) (107447) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:36,913::__init__::232::IOProcess::(_processLogs) (107447) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:37,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:37,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155716 s, 65.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:38,499::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:38,500::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69849::DEBUG::2014-09-25 17:13:38,502::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:39,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:39,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150488 s, 68.0 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 17:13:41,320::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:13:41,324::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,324::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,324::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,324::__init__::232::IOProcess::(_processLogs) (107448) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:13:41,325::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,325::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,325::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,326::__init__::232::IOProcess::(_processLogs) (107449) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:13:41,326::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,327::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,327::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,328::__init__::232::IOProcess::(_processLogs) (107450) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:13:41,328::__init__::232::IOProcess::(_processLogs) (107449) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:41,328::__init__::232::IOProcess::(_processLogs) (107448) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:41,328::__init__::232::IOProcess::(_processLogs) (107450) Queuing response >Thread-79::DEBUG::2014-09-25 17:13:41,333::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:13:41,334::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,334::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,334::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,335::__init__::232::IOProcess::(_processLogs) (107451) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:13:41,379::__init__::232::IOProcess::(_processLogs) (107451) Queuing response >Thread-79::DEBUG::2014-09-25 17:13:41,379::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:13:41,381::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,381::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,381::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,381::__init__::232::IOProcess::(_processLogs) (107452) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:13:41,398::__init__::232::IOProcess::(_processLogs) (107452) Queuing response >Thread-79::DEBUG::2014-09-25 17:13:41,398::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:13:41,400::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,401::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,401::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,401::__init__::232::IOProcess::(_processLogs) (107453) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:41,402::__init__::232::IOProcess::(_processLogs) (107453) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:41,402::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,402::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,403::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,403::__init__::232::IOProcess::(_processLogs) (107454) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:13:41,410::__init__::232::IOProcess::(_processLogs) (107454) Queuing response >Thread-79::DEBUG::2014-09-25 17:13:41,410::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 17:13:41,411::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:13:41,411::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:13:41,412::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,412::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,412::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,412::__init__::232::IOProcess::(_processLogs) (107455) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:41,415::__init__::232::IOProcess::(_processLogs) (107455) Queuing response >Thread-79::DEBUG::2014-09-25 17:13:41,416::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:13:41,432::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00693199 s, 81.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:41,433::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,433::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,433::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,434::__init__::232::IOProcess::(_processLogs) (107456) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:41,438::__init__::232::IOProcess::(_processLogs) (107456) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:41,440::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,440::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,440::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,441::__init__::232::IOProcess::(_processLogs) (107457) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:41,445::__init__::232::IOProcess::(_processLogs) (107457) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:41,446::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,446::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,446::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,446::__init__::232::IOProcess::(_processLogs) (107458) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:41,450::__init__::232::IOProcess::(_processLogs) (107458) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:41,451::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:41,451::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:41,452::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:41,452::__init__::232::IOProcess::(_processLogs) (107459) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:41,456::__init__::232::IOProcess::(_processLogs) (107459) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:41,516::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:41,517::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69853::DEBUG::2014-09-25 17:13:41,519::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:41,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:41,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170427 s, 60.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:42,509::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:42,509::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:42,510::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:42,510::__init__::232::IOProcess::(_processLogs) (107460) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:42,511::__init__::232::IOProcess::(_processLogs) (107460) Queuing response >Thread-80::DEBUG::2014-09-25 17:13:42,512::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:13:42,521::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000338955 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:42,522::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:42,523::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:42,523::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:42,525::__init__::232::IOProcess::(_processLogs) (107461) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:42,525::__init__::232::IOProcess::(_processLogs) (107461) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:43,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:43,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131513 s, 77.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:44,532::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:44,534::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69854::DEBUG::2014-09-25 17:13:44,536::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:45,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:45,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160661 s, 63.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:46,108::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:46,111::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69855::DEBUG::2014-09-25 17:13:46,112::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69855::DEBUG::2014-09-25 17:13:46,117::task::595::Storage.TaskManager.Task::(_updateState) Task=`698d4487-f168-4d3b-b918-0b4177497e7e`::moving from state init -> state preparing >Thread-69855::INFO::2014-09-25 17:13:46,118::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69855::INFO::2014-09-25 17:13:46,120::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69855::DEBUG::2014-09-25 17:13:46,122::task::1191::Storage.TaskManager.Task::(prepare) Task=`698d4487-f168-4d3b-b918-0b4177497e7e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69855::DEBUG::2014-09-25 17:13:46,123::task::595::Storage.TaskManager.Task::(_updateState) Task=`698d4487-f168-4d3b-b918-0b4177497e7e`::moving from state preparing -> state finished >Thread-69855::DEBUG::2014-09-25 17:13:46,123::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69855::DEBUG::2014-09-25 17:13:46,124::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69855::DEBUG::2014-09-25 17:13:46,126::task::993::Storage.TaskManager.Task::(_decref) Task=`698d4487-f168-4d3b-b918-0b4177497e7e`::ref 0 aborting False >Thread-69855::DEBUG::2014-09-25 17:13:46,127::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69855::DEBUG::2014-09-25 17:13:46,129::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:46,146::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:46,149::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69856::DEBUG::2014-09-25 17:13:46,149::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69856::DEBUG::2014-09-25 17:13:46,154::task::595::Storage.TaskManager.Task::(_updateState) Task=`3391ee0c-9d18-4511-84d0-c9414c37ae80`::moving from state init -> state preparing >Thread-69856::INFO::2014-09-25 17:13:46,155::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69856::DEBUG::2014-09-25 17:13:46,158::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`18e1cb81-0f82-4957-81f6-be8b31824ba0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69856::DEBUG::2014-09-25 17:13:46,159::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69856::DEBUG::2014-09-25 17:13:46,161::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69856::DEBUG::2014-09-25 17:13:46,162::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`18e1cb81-0f82-4957-81f6-be8b31824ba0`::Granted request >Thread-69856::DEBUG::2014-09-25 17:13:46,164::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3391ee0c-9d18-4511-84d0-c9414c37ae80`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69856::DEBUG::2014-09-25 17:13:46,165::task::993::Storage.TaskManager.Task::(_decref) Task=`3391ee0c-9d18-4511-84d0-c9414c37ae80`::ref 1 aborting False >Thread-69856::INFO::2014-09-25 17:13:46,185::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69856::DEBUG::2014-09-25 17:13:46,186::task::1191::Storage.TaskManager.Task::(prepare) Task=`3391ee0c-9d18-4511-84d0-c9414c37ae80`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69856::DEBUG::2014-09-25 17:13:46,187::task::595::Storage.TaskManager.Task::(_updateState) Task=`3391ee0c-9d18-4511-84d0-c9414c37ae80`::moving from state preparing -> state finished >Thread-69856::DEBUG::2014-09-25 17:13:46,188::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69856::DEBUG::2014-09-25 17:13:46,189::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69856::DEBUG::2014-09-25 17:13:46,190::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69856::DEBUG::2014-09-25 17:13:46,192::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69856::DEBUG::2014-09-25 17:13:46,193::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69856::DEBUG::2014-09-25 17:13:46,194::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69856::DEBUG::2014-09-25 17:13:46,196::task::993::Storage.TaskManager.Task::(_decref) Task=`3391ee0c-9d18-4511-84d0-c9414c37ae80`::ref 0 aborting False >Thread-69856::DEBUG::2014-09-25 17:13:46,198::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69856::DEBUG::2014-09-25 17:13:46,199::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:13:46,915::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:46,916::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:46,917::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:46,917::__init__::232::IOProcess::(_processLogs) (107462) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:46,921::__init__::232::IOProcess::(_processLogs) (107462) Queuing response >Thread-81::DEBUG::2014-09-25 17:13:46,923::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:13:46,936::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00500684 s, 74.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:46,938::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:46,938::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:46,939::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:46,940::__init__::232::IOProcess::(_processLogs) (107463) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:46,941::__init__::232::IOProcess::(_processLogs) (107463) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:47,548::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:47,549::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69857::DEBUG::2014-09-25 17:13:47,552::task::595::Storage.TaskManager.Task::(_updateState) Task=`1f3f0f29-84ea-4742-a342-0f0890a309f8`::moving from state init -> state preparing >Thread-69857::INFO::2014-09-25 17:13:47,552::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69857::INFO::2014-09-25 17:13:47,553::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00693199', 'lastCheck': '6.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338955', 'lastCheck': '5.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00500684', 'lastCheck': '0.6', 'valid': True}} >Thread-69857::DEBUG::2014-09-25 17:13:47,553::task::1191::Storage.TaskManager.Task::(prepare) Task=`1f3f0f29-84ea-4742-a342-0f0890a309f8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00693199', 'lastCheck': '6.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000338955', 'lastCheck': '5.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00500684', 'lastCheck': '0.6', 'valid': True}} >Thread-69857::DEBUG::2014-09-25 17:13:47,553::task::595::Storage.TaskManager.Task::(_updateState) Task=`1f3f0f29-84ea-4742-a342-0f0890a309f8`::moving from state preparing -> state finished >Thread-69857::DEBUG::2014-09-25 17:13:47,554::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69857::DEBUG::2014-09-25 17:13:47,554::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69857::DEBUG::2014-09-25 17:13:47,554::task::993::Storage.TaskManager.Task::(_decref) Task=`1f3f0f29-84ea-4742-a342-0f0890a309f8`::ref 0 aborting False >Thread-69857::DEBUG::2014-09-25 17:13:47,556::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:47,572::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:47,573::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69858::DEBUG::2014-09-25 17:13:47,574::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:47,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:47,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135684 s, 75.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:13:49,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:50,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0599185 s, 17.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:50,614::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:50,616::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69859::DEBUG::2014-09-25 17:13:50,618::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:13:51,457::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:51,458::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:51,458::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:51,458::__init__::232::IOProcess::(_processLogs) (107464) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:51,460::__init__::232::IOProcess::(_processLogs) (107464) Queuing response >Thread-79::DEBUG::2014-09-25 17:13:51,461::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:13:51,482::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00856534 s, 65.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:51,484::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:51,484::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:51,484::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:51,484::__init__::232::IOProcess::(_processLogs) (107465) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:51,490::__init__::232::IOProcess::(_processLogs) (107465) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:51,491::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:51,491::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:51,492::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:51,493::__init__::232::IOProcess::(_processLogs) (107466) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:51,493::__init__::232::IOProcess::(_processLogs) (107466) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:51,493::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:51,494::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:51,495::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:51,495::__init__::232::IOProcess::(_processLogs) (107467) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:51,495::__init__::232::IOProcess::(_processLogs) (107467) Queuing response >Thread-29::DEBUG::2014-09-25 17:13:51,496::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:51,496::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:51,496::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:51,497::__init__::232::IOProcess::(_processLogs) (107468) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:13:51,497::__init__::232::IOProcess::(_processLogs) (107468) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:52,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:13:52,526::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:52,527::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:52,527::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:52,527::__init__::232::IOProcess::(_processLogs) (107469) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:52,527::__init__::232::IOProcess::(_processLogs) (107469) Queuing response >Thread-80::DEBUG::2014-09-25 17:13:52,529::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:13:52,537::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000429136 s, 881 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:52,539::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:52,539::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:52,539::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:52,539::__init__::232::IOProcess::(_processLogs) (107470) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:52,540::__init__::232::IOProcess::(_processLogs) (107470) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:52,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0259716 s, 39.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:53,632::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:53,634::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69860::DEBUG::2014-09-25 17:13:53,636::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:54,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:54,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135328 s, 75.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:56,234::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69861::DEBUG::2014-09-25 17:13:56,236::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:13:56,236::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69861::DEBUG::2014-09-25 17:13:56,238::task::595::Storage.TaskManager.Task::(_updateState) Task=`42881d3c-2591-456b-a2e9-b1da699e0af0`::moving from state init -> state preparing >Thread-69861::INFO::2014-09-25 17:13:56,238::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69861::INFO::2014-09-25 17:13:56,239::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69861::DEBUG::2014-09-25 17:13:56,239::task::1191::Storage.TaskManager.Task::(prepare) Task=`42881d3c-2591-456b-a2e9-b1da699e0af0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69861::DEBUG::2014-09-25 17:13:56,240::task::595::Storage.TaskManager.Task::(_updateState) Task=`42881d3c-2591-456b-a2e9-b1da699e0af0`::moving from state preparing -> state finished >Thread-69861::DEBUG::2014-09-25 17:13:56,240::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69861::DEBUG::2014-09-25 17:13:56,240::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69861::DEBUG::2014-09-25 17:13:56,240::task::993::Storage.TaskManager.Task::(_decref) Task=`42881d3c-2591-456b-a2e9-b1da699e0af0`::ref 0 aborting False >Thread-69861::DEBUG::2014-09-25 17:13:56,241::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69861::DEBUG::2014-09-25 17:13:56,241::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:56,255::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69862::DEBUG::2014-09-25 17:13:56,257::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:13:56,256::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69862::DEBUG::2014-09-25 17:13:56,259::task::595::Storage.TaskManager.Task::(_updateState) Task=`937bbc0e-a1c3-4af0-88a0-3d771373d223`::moving from state init -> state preparing >Thread-69862::INFO::2014-09-25 17:13:56,259::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69862::DEBUG::2014-09-25 17:13:56,260::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`30dbf108-b624-4dc4-800e-34cab3fb6dc3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69862::DEBUG::2014-09-25 17:13:56,260::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69862::DEBUG::2014-09-25 17:13:56,260::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69862::DEBUG::2014-09-25 17:13:56,260::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`30dbf108-b624-4dc4-800e-34cab3fb6dc3`::Granted request >Thread-69862::DEBUG::2014-09-25 17:13:56,261::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`937bbc0e-a1c3-4af0-88a0-3d771373d223`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69862::DEBUG::2014-09-25 17:13:56,261::task::993::Storage.TaskManager.Task::(_decref) Task=`937bbc0e-a1c3-4af0-88a0-3d771373d223`::ref 1 aborting False >Thread-69862::INFO::2014-09-25 17:13:56,266::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69862::DEBUG::2014-09-25 17:13:56,266::task::1191::Storage.TaskManager.Task::(prepare) Task=`937bbc0e-a1c3-4af0-88a0-3d771373d223`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69862::DEBUG::2014-09-25 17:13:56,266::task::595::Storage.TaskManager.Task::(_updateState) Task=`937bbc0e-a1c3-4af0-88a0-3d771373d223`::moving from state preparing -> state finished >Thread-69862::DEBUG::2014-09-25 17:13:56,267::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69862::DEBUG::2014-09-25 17:13:56,267::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69862::DEBUG::2014-09-25 17:13:56,267::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69862::DEBUG::2014-09-25 17:13:56,267::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69862::DEBUG::2014-09-25 17:13:56,268::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69862::DEBUG::2014-09-25 17:13:56,268::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69862::DEBUG::2014-09-25 17:13:56,268::task::993::Storage.TaskManager.Task::(_decref) Task=`937bbc0e-a1c3-4af0-88a0-3d771373d223`::ref 0 aborting False >Thread-69862::DEBUG::2014-09-25 17:13:56,269::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69862::DEBUG::2014-09-25 17:13:56,269::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:13:56,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:56,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149725 s, 68.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:56,653::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:56,657::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69863::DEBUG::2014-09-25 17:13:56,661::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:13:56,943::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:56,943::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:56,944::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:56,944::__init__::232::IOProcess::(_processLogs) (107471) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:56,953::__init__::232::IOProcess::(_processLogs) (107471) Queuing response >Thread-81::DEBUG::2014-09-25 17:13:56,954::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:13:56,972::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00418678 s, 88.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:13:56,973::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:13:56,974::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:13:56,974::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:13:56,974::__init__::232::IOProcess::(_processLogs) (107472) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:13:56,976::__init__::232::IOProcess::(_processLogs) (107472) Queuing response >Dummy-148::DEBUG::2014-09-25 17:13:58,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:13:58,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148048 s, 69.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:13:59,677::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:13:59,679::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69864::DEBUG::2014-09-25 17:13:59,681::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:00,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:00,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141186 s, 72.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:01,498::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:01,498::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:01,498::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:01,499::__init__::232::IOProcess::(_processLogs) (107473) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:01,512::__init__::232::IOProcess::(_processLogs) (107473) Queuing response >Thread-79::DEBUG::2014-09-25 17:14:01,513::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:14:01,537::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00258407 s, 217 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:01,539::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:01,539::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:01,539::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:01,539::__init__::232::IOProcess::(_processLogs) (107474) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:01,545::__init__::232::IOProcess::(_processLogs) (107474) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:01,546::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:01,547::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:01,547::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:01,548::__init__::232::IOProcess::(_processLogs) (107475) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:01,548::__init__::232::IOProcess::(_processLogs) (107475) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:01,549::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:01,549::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:01,549::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:01,550::__init__::232::IOProcess::(_processLogs) (107476) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:01,550::__init__::232::IOProcess::(_processLogs) (107476) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:01,551::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:01,551::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:01,552::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:01,553::__init__::232::IOProcess::(_processLogs) (107477) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:01,553::__init__::232::IOProcess::(_processLogs) (107477) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:02,542::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:02,542::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:02,543::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:02,543::__init__::232::IOProcess::(_processLogs) (107478) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:02,544::__init__::232::IOProcess::(_processLogs) (107478) Queuing response >Thread-80::DEBUG::2014-09-25 17:14:02,545::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:14:02,554::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000385151 s, 981 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:02,555::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:02,556::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:02,557::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:02,558::__init__::232::IOProcess::(_processLogs) (107479) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:02,558::__init__::232::IOProcess::(_processLogs) (107479) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:02,695::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:02,697::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69865::DEBUG::2014-09-25 17:14:02,699::task::595::Storage.TaskManager.Task::(_updateState) Task=`76194fa5-a251-46b5-bfa2-0fa88fc8e3f5`::moving from state init -> state preparing >Thread-69865::INFO::2014-09-25 17:14:02,700::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69865::INFO::2014-09-25 17:14:02,700::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00258407', 'lastCheck': '1.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000385151', 'lastCheck': '0.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00418678', 'lastCheck': '5.7', 'valid': True}} >Thread-69865::DEBUG::2014-09-25 17:14:02,700::task::1191::Storage.TaskManager.Task::(prepare) Task=`76194fa5-a251-46b5-bfa2-0fa88fc8e3f5`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00258407', 'lastCheck': '1.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000385151', 'lastCheck': '0.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00418678', 'lastCheck': '5.7', 'valid': True}} >Thread-69865::DEBUG::2014-09-25 17:14:02,701::task::595::Storage.TaskManager.Task::(_updateState) Task=`76194fa5-a251-46b5-bfa2-0fa88fc8e3f5`::moving from state preparing -> state finished >Thread-69865::DEBUG::2014-09-25 17:14:02,701::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69865::DEBUG::2014-09-25 17:14:02,701::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69865::DEBUG::2014-09-25 17:14:02,701::task::993::Storage.TaskManager.Task::(_decref) Task=`76194fa5-a251-46b5-bfa2-0fa88fc8e3f5`::ref 0 aborting False >Thread-69865::DEBUG::2014-09-25 17:14:02,704::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:02,726::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:02,727::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69866::DEBUG::2014-09-25 17:14:02,729::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:02,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:02,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143208 s, 71.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:14:04,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:04,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0176608 s, 58.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:05,784::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:05,786::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69867::DEBUG::2014-09-25 17:14:05,788::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:06,299::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:06,300::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69868::DEBUG::2014-09-25 17:14:06,300::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69868::DEBUG::2014-09-25 17:14:06,302::task::595::Storage.TaskManager.Task::(_updateState) Task=`eaf97047-0492-4139-a971-90c9554403a1`::moving from state init -> state preparing >Thread-69868::INFO::2014-09-25 17:14:06,302::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69868::INFO::2014-09-25 17:14:06,303::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69868::DEBUG::2014-09-25 17:14:06,303::task::1191::Storage.TaskManager.Task::(prepare) Task=`eaf97047-0492-4139-a971-90c9554403a1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69868::DEBUG::2014-09-25 17:14:06,303::task::595::Storage.TaskManager.Task::(_updateState) Task=`eaf97047-0492-4139-a971-90c9554403a1`::moving from state preparing -> state finished >Thread-69868::DEBUG::2014-09-25 17:14:06,303::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69868::DEBUG::2014-09-25 17:14:06,303::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69868::DEBUG::2014-09-25 17:14:06,304::task::993::Storage.TaskManager.Task::(_decref) Task=`eaf97047-0492-4139-a971-90c9554403a1`::ref 0 aborting False >Thread-69868::DEBUG::2014-09-25 17:14:06,304::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69868::DEBUG::2014-09-25 17:14:06,304::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:06,318::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69869::DEBUG::2014-09-25 17:14:06,319::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:14:06,319::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69869::DEBUG::2014-09-25 17:14:06,321::task::595::Storage.TaskManager.Task::(_updateState) Task=`5307ca02-1a3d-4990-b638-2f4f2215d5a8`::moving from state init -> state preparing >Thread-69869::INFO::2014-09-25 17:14:06,321::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69869::DEBUG::2014-09-25 17:14:06,322::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`44586bde-5866-4ceb-a818-6938c9ca984b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69869::DEBUG::2014-09-25 17:14:06,322::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69869::DEBUG::2014-09-25 17:14:06,323::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69869::DEBUG::2014-09-25 17:14:06,323::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`44586bde-5866-4ceb-a818-6938c9ca984b`::Granted request >Thread-69869::DEBUG::2014-09-25 17:14:06,323::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5307ca02-1a3d-4990-b638-2f4f2215d5a8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69869::DEBUG::2014-09-25 17:14:06,324::task::993::Storage.TaskManager.Task::(_decref) Task=`5307ca02-1a3d-4990-b638-2f4f2215d5a8`::ref 1 aborting False >Thread-69869::INFO::2014-09-25 17:14:06,328::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69869::DEBUG::2014-09-25 17:14:06,328::task::1191::Storage.TaskManager.Task::(prepare) Task=`5307ca02-1a3d-4990-b638-2f4f2215d5a8`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69869::DEBUG::2014-09-25 17:14:06,329::task::595::Storage.TaskManager.Task::(_updateState) Task=`5307ca02-1a3d-4990-b638-2f4f2215d5a8`::moving from state preparing -> state finished >Thread-69869::DEBUG::2014-09-25 17:14:06,329::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69869::DEBUG::2014-09-25 17:14:06,329::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69869::DEBUG::2014-09-25 17:14:06,329::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69869::DEBUG::2014-09-25 17:14:06,330::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69869::DEBUG::2014-09-25 17:14:06,330::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69869::DEBUG::2014-09-25 17:14:06,330::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69869::DEBUG::2014-09-25 17:14:06,331::task::993::Storage.TaskManager.Task::(_decref) Task=`5307ca02-1a3d-4990-b638-2f4f2215d5a8`::ref 0 aborting False >Thread-69869::DEBUG::2014-09-25 17:14:06,331::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69869::DEBUG::2014-09-25 17:14:06,331::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:06,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:06,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133046 s, 77.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:06,978::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:06,979::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:06,979::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:06,979::__init__::232::IOProcess::(_processLogs) (107480) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:06,980::__init__::232::IOProcess::(_processLogs) (107480) Queuing response >Thread-81::DEBUG::2014-09-25 17:14:06,981::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:14:06,990::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000373449 s, 993 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:06,991::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:06,991::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:06,992::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:06,992::__init__::232::IOProcess::(_processLogs) (107481) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:06,992::__init__::232::IOProcess::(_processLogs) (107481) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:08,805::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:08,807::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69870::DEBUG::2014-09-25 17:14:08,809::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:08,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:08,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016827 s, 60.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:14:10,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:11,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.226819 s, 4.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:11,553::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:11,554::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:11,554::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:11,554::__init__::232::IOProcess::(_processLogs) (107482) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:11,565::__init__::232::IOProcess::(_processLogs) (107482) Queuing response >Thread-79::DEBUG::2014-09-25 17:14:11,565::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:14:11,579::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00215275 s, 261 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:11,580::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:11,580::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:11,581::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:11,581::__init__::232::IOProcess::(_processLogs) (107483) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:11,586::__init__::232::IOProcess::(_processLogs) (107483) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:11,588::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:11,588::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:11,589::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:11,589::__init__::232::IOProcess::(_processLogs) (107484) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:11,589::__init__::232::IOProcess::(_processLogs) (107484) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:11,590::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:11,590::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:11,591::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:11,591::__init__::232::IOProcess::(_processLogs) (107485) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:11,592::__init__::232::IOProcess::(_processLogs) (107485) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:11,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:11,592::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:11,593::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:11,593::__init__::232::IOProcess::(_processLogs) (107486) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:11,593::__init__::232::IOProcess::(_processLogs) (107486) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:11,822::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:11,824::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69871::DEBUG::2014-09-25 17:14:11,826::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-80::DEBUG::2014-09-25 17:14:12,558::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:14:12,561::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:12,562::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:12,562::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:12,562::__init__::232::IOProcess::(_processLogs) (107487) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:14:12,563::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:12,563::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:12,563::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:12,564::__init__::232::IOProcess::(_processLogs) (107488) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:14:12,564::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:12,566::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:12,568::__init__::232::IOProcess::(_processLogs) (107488) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:12,570::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:12,571::__init__::232::IOProcess::(_processLogs) (107489) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:14:12,574::__init__::232::IOProcess::(_processLogs) (107487) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:12,576::__init__::232::IOProcess::(_processLogs) (107489) Queuing response >Thread-80::DEBUG::2014-09-25 17:14:12,579::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:14:12,579::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:12,580::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:12,580::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:12,580::__init__::232::IOProcess::(_processLogs) (107490) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:14:12,583::__init__::232::IOProcess::(_processLogs) (107490) Queuing response >Thread-80::DEBUG::2014-09-25 17:14:12,584::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:14:12,584::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:12,585::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:12,585::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:12,585::__init__::232::IOProcess::(_processLogs) (107491) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:14:12,586::__init__::232::IOProcess::(_processLogs) (107491) Queuing response >Thread-80::DEBUG::2014-09-25 17:14:12,586::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:14:12,587::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:12,588::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:12,588::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:12,588::__init__::232::IOProcess::(_processLogs) (107492) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:12,588::__init__::232::IOProcess::(_processLogs) (107492) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:12,589::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:12,589::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:12,590::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:12,590::__init__::232::IOProcess::(_processLogs) (107493) Got request for method 'glob' >Thread-80::DEBUG::2014-09-25 17:14:12,591::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:14:12,591::__init__::232::IOProcess::(_processLogs) (107493) Queuing response >Thread-80::INFO::2014-09-25 17:14:12,591::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:14:12,592::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:14:12,593::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:12,593::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:12,593::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:12,593::__init__::232::IOProcess::(_processLogs) (107494) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:12,594::__init__::232::IOProcess::(_processLogs) (107494) Queuing response >Thread-80::DEBUG::2014-09-25 17:14:12,595::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:14:12,604::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000350897 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:12,605::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:12,606::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:12,606::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:12,606::__init__::232::IOProcess::(_processLogs) (107495) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:12,607::__init__::232::IOProcess::(_processLogs) (107495) Queuing response >Dummy-148::DEBUG::2014-09-25 17:14:13,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:13,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137573 s, 74.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:14,837::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:14,839::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69875::DEBUG::2014-09-25 17:14:14,841::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:15,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:15,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020058 s, 51.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:16,363::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69876::DEBUG::2014-09-25 17:14:16,367::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:14:16,367::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69876::DEBUG::2014-09-25 17:14:16,372::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd998180-7170-4201-87a7-fdda141a5e2b`::moving from state init -> state preparing >Thread-69876::INFO::2014-09-25 17:14:16,373::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69876::INFO::2014-09-25 17:14:16,375::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69876::DEBUG::2014-09-25 17:14:16,376::task::1191::Storage.TaskManager.Task::(prepare) Task=`dd998180-7170-4201-87a7-fdda141a5e2b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69876::DEBUG::2014-09-25 17:14:16,377::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd998180-7170-4201-87a7-fdda141a5e2b`::moving from state preparing -> state finished >Thread-69876::DEBUG::2014-09-25 17:14:16,378::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69876::DEBUG::2014-09-25 17:14:16,378::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69876::DEBUG::2014-09-25 17:14:16,379::task::993::Storage.TaskManager.Task::(_decref) Task=`dd998180-7170-4201-87a7-fdda141a5e2b`::ref 0 aborting False >Thread-69876::DEBUG::2014-09-25 17:14:16,381::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69876::DEBUG::2014-09-25 17:14:16,382::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:16,395::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69877::DEBUG::2014-09-25 17:14:16,398::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:14:16,398::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69877::DEBUG::2014-09-25 17:14:16,402::task::595::Storage.TaskManager.Task::(_updateState) Task=`a2e265dc-e028-4920-a487-17be136fa0df`::moving from state init -> state preparing >Thread-69877::INFO::2014-09-25 17:14:16,403::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69877::DEBUG::2014-09-25 17:14:16,406::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`351d0d5f-40f4-4529-9218-10512127c187`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69877::DEBUG::2014-09-25 17:14:16,406::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69877::DEBUG::2014-09-25 17:14:16,408::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69877::DEBUG::2014-09-25 17:14:16,409::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`351d0d5f-40f4-4529-9218-10512127c187`::Granted request >Thread-69877::DEBUG::2014-09-25 17:14:16,411::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a2e265dc-e028-4920-a487-17be136fa0df`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69877::DEBUG::2014-09-25 17:14:16,412::task::993::Storage.TaskManager.Task::(_decref) Task=`a2e265dc-e028-4920-a487-17be136fa0df`::ref 1 aborting False >Thread-69877::INFO::2014-09-25 17:14:16,429::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69877::DEBUG::2014-09-25 17:14:16,430::task::1191::Storage.TaskManager.Task::(prepare) Task=`a2e265dc-e028-4920-a487-17be136fa0df`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69877::DEBUG::2014-09-25 17:14:16,431::task::595::Storage.TaskManager.Task::(_updateState) Task=`a2e265dc-e028-4920-a487-17be136fa0df`::moving from state preparing -> state finished >Thread-69877::DEBUG::2014-09-25 17:14:16,431::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69877::DEBUG::2014-09-25 17:14:16,432::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69877::DEBUG::2014-09-25 17:14:16,434::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69877::DEBUG::2014-09-25 17:14:16,435::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69877::DEBUG::2014-09-25 17:14:16,436::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69877::DEBUG::2014-09-25 17:14:16,436::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69877::DEBUG::2014-09-25 17:14:16,438::task::993::Storage.TaskManager.Task::(_decref) Task=`a2e265dc-e028-4920-a487-17be136fa0df`::ref 0 aborting False >Thread-69877::DEBUG::2014-09-25 17:14:16,440::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69877::DEBUG::2014-09-25 17:14:16,441::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:14:16,996::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:16,997::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:16,999::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:17,000::__init__::232::IOProcess::(_processLogs) (107496) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:17,002::__init__::232::IOProcess::(_processLogs) (107496) Queuing response >Thread-81::DEBUG::2014-09-25 17:14:17,004::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:14:17,024::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00391879 s, 94.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:17,025::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:17,025::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:17,025::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:17,026::__init__::232::IOProcess::(_processLogs) (107497) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:17,028::__init__::232::IOProcess::(_processLogs) (107497) Queuing response >Dummy-148::DEBUG::2014-09-25 17:14:17,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:17,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159752 s, 64.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:17,855::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:17,856::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69878::DEBUG::2014-09-25 17:14:17,859::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d72eee1-ad4b-4610-ac0e-6d0f80918fb2`::moving from state init -> state preparing >Thread-69878::INFO::2014-09-25 17:14:17,860::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69878::INFO::2014-09-25 17:14:17,860::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00215275', 'lastCheck': '6.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000350897', 'lastCheck': '5.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00391879', 'lastCheck': '0.8', 'valid': True}} >Thread-69878::DEBUG::2014-09-25 17:14:17,860::task::1191::Storage.TaskManager.Task::(prepare) Task=`7d72eee1-ad4b-4610-ac0e-6d0f80918fb2`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00215275', 'lastCheck': '6.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000350897', 'lastCheck': '5.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00391879', 'lastCheck': '0.8', 'valid': True}} >Thread-69878::DEBUG::2014-09-25 17:14:17,861::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d72eee1-ad4b-4610-ac0e-6d0f80918fb2`::moving from state preparing -> state finished >Thread-69878::DEBUG::2014-09-25 17:14:17,861::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69878::DEBUG::2014-09-25 17:14:17,861::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69878::DEBUG::2014-09-25 17:14:17,861::task::993::Storage.TaskManager.Task::(_decref) Task=`7d72eee1-ad4b-4610-ac0e-6d0f80918fb2`::ref 0 aborting False >Thread-69878::DEBUG::2014-09-25 17:14:17,864::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:17,885::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:17,887::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69879::DEBUG::2014-09-25 17:14:17,889::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:19,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:19,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0196564 s, 52.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:20,957::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:20,958::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69880::DEBUG::2014-09-25 17:14:20,960::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:21,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:21,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0381987 s, 26.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:21,595::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:21,595::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:21,595::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:21,595::__init__::232::IOProcess::(_processLogs) (107498) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:21,596::__init__::232::IOProcess::(_processLogs) (107498) Queuing response >Thread-79::DEBUG::2014-09-25 17:14:21,597::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:14:21,610::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00285581 s, 197 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:21,611::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:21,611::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:21,611::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:21,612::__init__::232::IOProcess::(_processLogs) (107499) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:21,615::__init__::232::IOProcess::(_processLogs) (107499) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:21,617::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:21,617::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:21,617::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:21,618::__init__::232::IOProcess::(_processLogs) (107500) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:21,618::__init__::232::IOProcess::(_processLogs) (107500) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:21,619::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:21,619::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:21,620::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:21,620::__init__::232::IOProcess::(_processLogs) (107501) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:21,620::__init__::232::IOProcess::(_processLogs) (107501) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:21,621::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:21,621::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:21,622::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:21,623::__init__::232::IOProcess::(_processLogs) (107502) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:21,623::__init__::232::IOProcess::(_processLogs) (107502) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:22,609::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:22,609::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:22,609::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:22,609::__init__::232::IOProcess::(_processLogs) (107503) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:22,610::__init__::232::IOProcess::(_processLogs) (107503) Queuing response >Thread-80::DEBUG::2014-09-25 17:14:22,611::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:14:22,620::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000536251 s, 705 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:22,621::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:22,622::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:22,622::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:22,622::__init__::232::IOProcess::(_processLogs) (107504) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:22,623::__init__::232::IOProcess::(_processLogs) (107504) Queuing response >Dummy-148::DEBUG::2014-09-25 17:14:23,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:23,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183249 s, 55.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:23,971::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:23,973::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69881::DEBUG::2014-09-25 17:14:23,975::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:25,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:25,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019413 s, 52.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:26,468::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69882::DEBUG::2014-09-25 17:14:26,469::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:14:26,469::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69882::DEBUG::2014-09-25 17:14:26,471::task::595::Storage.TaskManager.Task::(_updateState) Task=`9caa8d39-a597-4b81-ad14-cef82f3cfa04`::moving from state init -> state preparing >Thread-69882::INFO::2014-09-25 17:14:26,472::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69882::INFO::2014-09-25 17:14:26,472::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69882::DEBUG::2014-09-25 17:14:26,473::task::1191::Storage.TaskManager.Task::(prepare) Task=`9caa8d39-a597-4b81-ad14-cef82f3cfa04`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69882::DEBUG::2014-09-25 17:14:26,473::task::595::Storage.TaskManager.Task::(_updateState) Task=`9caa8d39-a597-4b81-ad14-cef82f3cfa04`::moving from state preparing -> state finished >Thread-69882::DEBUG::2014-09-25 17:14:26,473::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69882::DEBUG::2014-09-25 17:14:26,474::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69882::DEBUG::2014-09-25 17:14:26,474::task::993::Storage.TaskManager.Task::(_decref) Task=`9caa8d39-a597-4b81-ad14-cef82f3cfa04`::ref 0 aborting False >Thread-69882::DEBUG::2014-09-25 17:14:26,475::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69882::DEBUG::2014-09-25 17:14:26,475::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:26,486::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:26,487::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69883::DEBUG::2014-09-25 17:14:26,487::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69883::DEBUG::2014-09-25 17:14:26,489::task::595::Storage.TaskManager.Task::(_updateState) Task=`a330b6a5-c263-46be-8680-3b094ebfe69e`::moving from state init -> state preparing >Thread-69883::INFO::2014-09-25 17:14:26,489::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69883::DEBUG::2014-09-25 17:14:26,490::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`25f51576-8c31-4bab-ad1f-f6da1934bf8c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69883::DEBUG::2014-09-25 17:14:26,490::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69883::DEBUG::2014-09-25 17:14:26,490::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69883::DEBUG::2014-09-25 17:14:26,491::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`25f51576-8c31-4bab-ad1f-f6da1934bf8c`::Granted request >Thread-69883::DEBUG::2014-09-25 17:14:26,491::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a330b6a5-c263-46be-8680-3b094ebfe69e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69883::DEBUG::2014-09-25 17:14:26,491::task::993::Storage.TaskManager.Task::(_decref) Task=`a330b6a5-c263-46be-8680-3b094ebfe69e`::ref 1 aborting False >Thread-69883::INFO::2014-09-25 17:14:26,496::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69883::DEBUG::2014-09-25 17:14:26,496::task::1191::Storage.TaskManager.Task::(prepare) Task=`a330b6a5-c263-46be-8680-3b094ebfe69e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69883::DEBUG::2014-09-25 17:14:26,496::task::595::Storage.TaskManager.Task::(_updateState) Task=`a330b6a5-c263-46be-8680-3b094ebfe69e`::moving from state preparing -> state finished >Thread-69883::DEBUG::2014-09-25 17:14:26,497::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69883::DEBUG::2014-09-25 17:14:26,497::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69883::DEBUG::2014-09-25 17:14:26,497::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69883::DEBUG::2014-09-25 17:14:26,497::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69883::DEBUG::2014-09-25 17:14:26,498::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69883::DEBUG::2014-09-25 17:14:26,498::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69883::DEBUG::2014-09-25 17:14:26,498::task::993::Storage.TaskManager.Task::(_decref) Task=`a330b6a5-c263-46be-8680-3b094ebfe69e`::ref 0 aborting False >Thread-69883::DEBUG::2014-09-25 17:14:26,499::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69883::DEBUG::2014-09-25 17:14:26,499::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:26,986::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:26,987::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69884::DEBUG::2014-09-25 17:14:26,989::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:14:27,029::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:27,030::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:27,030::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:27,030::__init__::232::IOProcess::(_processLogs) (107505) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:27,033::__init__::232::IOProcess::(_processLogs) (107505) Queuing response >Thread-81::DEBUG::2014-09-25 17:14:27,034::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:14:27,050::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00389138 s, 95.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:27,051::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:27,052::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:27,052::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:27,052::__init__::232::IOProcess::(_processLogs) (107506) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:27,055::__init__::232::IOProcess::(_processLogs) (107506) Queuing response >Dummy-148::DEBUG::2014-09-25 17:14:27,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:27,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193441 s, 52.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:14:29,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:29,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214687 s, 47.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:30,000::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:30,001::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69885::DEBUG::2014-09-25 17:14:30,003::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:14:31,624::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:31,624::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:31,624::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:31,624::__init__::232::IOProcess::(_processLogs) (107507) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:31,626::__init__::232::IOProcess::(_processLogs) (107507) Queuing response >Thread-79::DEBUG::2014-09-25 17:14:31,627::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:14:31,643::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00612737 s, 91.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:31,644::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:31,644::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:31,645::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:31,645::__init__::232::IOProcess::(_processLogs) (107508) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:31,648::__init__::232::IOProcess::(_processLogs) (107508) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:31,649::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:31,649::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:31,649::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:31,650::__init__::232::IOProcess::(_processLogs) (107509) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:31,650::__init__::232::IOProcess::(_processLogs) (107509) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:31,651::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:31,651::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:31,651::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:31,652::__init__::232::IOProcess::(_processLogs) (107510) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:31,652::__init__::232::IOProcess::(_processLogs) (107510) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:31,653::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:31,653::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:31,653::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:31,653::__init__::232::IOProcess::(_processLogs) (107511) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:31,654::__init__::232::IOProcess::(_processLogs) (107511) Queuing response >Dummy-148::DEBUG::2014-09-25 17:14:31,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:31,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016665 s, 61.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:32,625::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:32,625::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:32,626::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:32,626::__init__::232::IOProcess::(_processLogs) (107512) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:32,626::__init__::232::IOProcess::(_processLogs) (107512) Queuing response >Thread-80::DEBUG::2014-09-25 17:14:32,628::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:14:32,636::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000489635 s, 772 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:32,637::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:32,638::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:32,638::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:32,638::__init__::232::IOProcess::(_processLogs) (107513) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:32,639::__init__::232::IOProcess::(_processLogs) (107513) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:33,015::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:33,016::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69886::DEBUG::2014-09-25 17:14:33,020::task::595::Storage.TaskManager.Task::(_updateState) Task=`bbe5ae14-c14f-4405-af80-62a24b832799`::moving from state init -> state preparing >Thread-69886::INFO::2014-09-25 17:14:33,020::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69886::INFO::2014-09-25 17:14:33,020::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00612737', 'lastCheck': '1.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000489635', 'lastCheck': '0.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00389138', 'lastCheck': '6.0', 'valid': True}} >Thread-69886::DEBUG::2014-09-25 17:14:33,021::task::1191::Storage.TaskManager.Task::(prepare) Task=`bbe5ae14-c14f-4405-af80-62a24b832799`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00612737', 'lastCheck': '1.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000489635', 'lastCheck': '0.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00389138', 'lastCheck': '6.0', 'valid': True}} >Thread-69886::DEBUG::2014-09-25 17:14:33,021::task::595::Storage.TaskManager.Task::(_updateState) Task=`bbe5ae14-c14f-4405-af80-62a24b832799`::moving from state preparing -> state finished >Thread-69886::DEBUG::2014-09-25 17:14:33,021::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69886::DEBUG::2014-09-25 17:14:33,021::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69886::DEBUG::2014-09-25 17:14:33,022::task::993::Storage.TaskManager.Task::(_decref) Task=`bbe5ae14-c14f-4405-af80-62a24b832799`::ref 0 aborting False >Thread-69886::DEBUG::2014-09-25 17:14:33,024::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:33,044::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:33,045::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69887::DEBUG::2014-09-25 17:14:33,047::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:33,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:33,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213929 s, 47.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:14:35,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:35,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214231 s, 47.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:36,103::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:36,104::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69888::DEBUG::2014-09-25 17:14:36,106::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:36,522::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:36,523::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69889::DEBUG::2014-09-25 17:14:36,523::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69889::DEBUG::2014-09-25 17:14:36,525::task::595::Storage.TaskManager.Task::(_updateState) Task=`06c10747-2ecc-457a-933e-7296a085e43d`::moving from state init -> state preparing >Thread-69889::INFO::2014-09-25 17:14:36,525::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69889::INFO::2014-09-25 17:14:36,526::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69889::DEBUG::2014-09-25 17:14:36,526::task::1191::Storage.TaskManager.Task::(prepare) Task=`06c10747-2ecc-457a-933e-7296a085e43d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69889::DEBUG::2014-09-25 17:14:36,526::task::595::Storage.TaskManager.Task::(_updateState) Task=`06c10747-2ecc-457a-933e-7296a085e43d`::moving from state preparing -> state finished >Thread-69889::DEBUG::2014-09-25 17:14:36,527::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69889::DEBUG::2014-09-25 17:14:36,527::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69889::DEBUG::2014-09-25 17:14:36,527::task::993::Storage.TaskManager.Task::(_decref) Task=`06c10747-2ecc-457a-933e-7296a085e43d`::ref 0 aborting False >Thread-69889::DEBUG::2014-09-25 17:14:36,527::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69889::DEBUG::2014-09-25 17:14:36,528::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:36,539::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69890::DEBUG::2014-09-25 17:14:36,540::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:14:36,540::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69890::DEBUG::2014-09-25 17:14:36,542::task::595::Storage.TaskManager.Task::(_updateState) Task=`fbc9bc88-511a-4dde-9526-d2d208f1d95b`::moving from state init -> state preparing >Thread-69890::INFO::2014-09-25 17:14:36,542::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69890::DEBUG::2014-09-25 17:14:36,543::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cd029fc2-46c7-4401-b227-b719c2262442`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69890::DEBUG::2014-09-25 17:14:36,543::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69890::DEBUG::2014-09-25 17:14:36,544::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69890::DEBUG::2014-09-25 17:14:36,544::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cd029fc2-46c7-4401-b227-b719c2262442`::Granted request >Thread-69890::DEBUG::2014-09-25 17:14:36,545::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`fbc9bc88-511a-4dde-9526-d2d208f1d95b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69890::DEBUG::2014-09-25 17:14:36,545::task::993::Storage.TaskManager.Task::(_decref) Task=`fbc9bc88-511a-4dde-9526-d2d208f1d95b`::ref 1 aborting False >Thread-69890::INFO::2014-09-25 17:14:36,549::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69890::DEBUG::2014-09-25 17:14:36,549::task::1191::Storage.TaskManager.Task::(prepare) Task=`fbc9bc88-511a-4dde-9526-d2d208f1d95b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69890::DEBUG::2014-09-25 17:14:36,550::task::595::Storage.TaskManager.Task::(_updateState) Task=`fbc9bc88-511a-4dde-9526-d2d208f1d95b`::moving from state preparing -> state finished >Thread-69890::DEBUG::2014-09-25 17:14:36,550::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69890::DEBUG::2014-09-25 17:14:36,550::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69890::DEBUG::2014-09-25 17:14:36,551::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69890::DEBUG::2014-09-25 17:14:36,551::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69890::DEBUG::2014-09-25 17:14:36,551::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69890::DEBUG::2014-09-25 17:14:36,551::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69890::DEBUG::2014-09-25 17:14:36,552::task::993::Storage.TaskManager.Task::(_decref) Task=`fbc9bc88-511a-4dde-9526-d2d208f1d95b`::ref 0 aborting False >Thread-69890::DEBUG::2014-09-25 17:14:36,552::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69890::DEBUG::2014-09-25 17:14:36,553::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:14:37,057::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:37,057::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:37,058::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:37,058::__init__::232::IOProcess::(_processLogs) (107514) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:37,061::__init__::232::IOProcess::(_processLogs) (107514) Queuing response >Thread-81::DEBUG::2014-09-25 17:14:37,062::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:14:37,077::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00464787 s, 79.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:37,079::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:37,079::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:37,079::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:37,079::__init__::232::IOProcess::(_processLogs) (107515) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:37,080::__init__::232::IOProcess::(_processLogs) (107515) Queuing response >Dummy-148::DEBUG::2014-09-25 17:14:37,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:38,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0487813 s, 21.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:39,120::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:39,122::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69891::DEBUG::2014-09-25 17:14:39,124::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:40,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:40,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0198179 s, 51.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:41,655::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:41,656::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:41,656::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:41,656::__init__::232::IOProcess::(_processLogs) (107516) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:41,656::__init__::232::IOProcess::(_processLogs) (107516) Queuing response >Thread-79::DEBUG::2014-09-25 17:14:41,658::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:14:41,666::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000431941 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:41,667::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:41,668::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:41,668::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:41,668::__init__::232::IOProcess::(_processLogs) (107517) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:41,669::__init__::232::IOProcess::(_processLogs) (107517) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:41,670::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:41,671::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:41,671::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:41,671::__init__::232::IOProcess::(_processLogs) (107518) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:41,672::__init__::232::IOProcess::(_processLogs) (107518) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:41,672::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:41,673::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:41,673::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:41,673::__init__::232::IOProcess::(_processLogs) (107519) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:41,674::__init__::232::IOProcess::(_processLogs) (107519) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:41,675::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:41,675::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:41,675::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:41,675::__init__::232::IOProcess::(_processLogs) (107520) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:41,676::__init__::232::IOProcess::(_processLogs) (107520) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:42,138::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:42,140::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69892::DEBUG::2014-09-25 17:14:42,142::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:42,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:42,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015333 s, 66.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:42,641::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:42,641::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:42,642::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:42,642::__init__::232::IOProcess::(_processLogs) (107521) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:42,643::__init__::232::IOProcess::(_processLogs) (107521) Queuing response >Thread-80::DEBUG::2014-09-25 17:14:42,644::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:14:42,654::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000456287 s, 828 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:42,655::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:42,655::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:42,655::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:42,656::__init__::232::IOProcess::(_processLogs) (107522) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:42,656::__init__::232::IOProcess::(_processLogs) (107522) Queuing response >Dummy-148::DEBUG::2014-09-25 17:14:44,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:44,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178324 s, 57.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:45,155::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:45,157::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69893::DEBUG::2014-09-25 17:14:45,159::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:46,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:46,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0257259 s, 39.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:46,579::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:46,582::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69894::DEBUG::2014-09-25 17:14:46,582::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69894::DEBUG::2014-09-25 17:14:46,588::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3b35a58-86b5-42d6-8d51-7fa16e5be522`::moving from state init -> state preparing >Thread-69894::INFO::2014-09-25 17:14:46,589::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69894::INFO::2014-09-25 17:14:46,591::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69894::DEBUG::2014-09-25 17:14:46,592::task::1191::Storage.TaskManager.Task::(prepare) Task=`e3b35a58-86b5-42d6-8d51-7fa16e5be522`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69894::DEBUG::2014-09-25 17:14:46,593::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3b35a58-86b5-42d6-8d51-7fa16e5be522`::moving from state preparing -> state finished >Thread-69894::DEBUG::2014-09-25 17:14:46,594::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69894::DEBUG::2014-09-25 17:14:46,595::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69894::DEBUG::2014-09-25 17:14:46,597::task::993::Storage.TaskManager.Task::(_decref) Task=`e3b35a58-86b5-42d6-8d51-7fa16e5be522`::ref 0 aborting False >Thread-69894::DEBUG::2014-09-25 17:14:46,599::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69894::DEBUG::2014-09-25 17:14:46,602::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:46,619::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:46,623::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69895::DEBUG::2014-09-25 17:14:46,623::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69895::DEBUG::2014-09-25 17:14:46,628::task::595::Storage.TaskManager.Task::(_updateState) Task=`57d590af-cb86-4ddd-9df6-dbc0e099791e`::moving from state init -> state preparing >Thread-69895::INFO::2014-09-25 17:14:46,629::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69895::DEBUG::2014-09-25 17:14:46,632::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c7b7d7b0-eac5-4000-b10a-da1569d692ef`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69895::DEBUG::2014-09-25 17:14:46,633::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69895::DEBUG::2014-09-25 17:14:46,635::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69895::DEBUG::2014-09-25 17:14:46,636::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c7b7d7b0-eac5-4000-b10a-da1569d692ef`::Granted request >Thread-69895::DEBUG::2014-09-25 17:14:46,639::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`57d590af-cb86-4ddd-9df6-dbc0e099791e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69895::DEBUG::2014-09-25 17:14:46,640::task::993::Storage.TaskManager.Task::(_decref) Task=`57d590af-cb86-4ddd-9df6-dbc0e099791e`::ref 1 aborting False >Thread-69895::INFO::2014-09-25 17:14:46,659::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69895::DEBUG::2014-09-25 17:14:46,660::task::1191::Storage.TaskManager.Task::(prepare) Task=`57d590af-cb86-4ddd-9df6-dbc0e099791e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69895::DEBUG::2014-09-25 17:14:46,661::task::595::Storage.TaskManager.Task::(_updateState) Task=`57d590af-cb86-4ddd-9df6-dbc0e099791e`::moving from state preparing -> state finished >Thread-69895::DEBUG::2014-09-25 17:14:46,662::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69895::DEBUG::2014-09-25 17:14:46,663::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69895::DEBUG::2014-09-25 17:14:46,664::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69895::DEBUG::2014-09-25 17:14:46,666::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69895::DEBUG::2014-09-25 17:14:46,667::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69895::DEBUG::2014-09-25 17:14:46,668::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69895::DEBUG::2014-09-25 17:14:46,670::task::993::Storage.TaskManager.Task::(_decref) Task=`57d590af-cb86-4ddd-9df6-dbc0e099791e`::ref 0 aborting False >Thread-69895::DEBUG::2014-09-25 17:14:46,671::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69895::DEBUG::2014-09-25 17:14:46,673::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:14:47,082::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:47,083::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:47,083::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:47,083::__init__::232::IOProcess::(_processLogs) (107523) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:47,090::__init__::232::IOProcess::(_processLogs) (107523) Queuing response >Thread-81::DEBUG::2014-09-25 17:14:47,090::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:14:47,105::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00294033 s, 126 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:47,106::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:47,106::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:47,107::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:47,107::__init__::232::IOProcess::(_processLogs) (107524) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:47,111::__init__::232::IOProcess::(_processLogs) (107524) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:48,173::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:48,175::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69896::DEBUG::2014-09-25 17:14:48,177::task::595::Storage.TaskManager.Task::(_updateState) Task=`317b1722-f8ee-4daa-9939-52eb03a0daa2`::moving from state init -> state preparing >Thread-69896::INFO::2014-09-25 17:14:48,178::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69896::INFO::2014-09-25 17:14:48,178::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000431941', 'lastCheck': '6.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000456287', 'lastCheck': '5.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00294033', 'lastCheck': '1.1', 'valid': True}} >Thread-69896::DEBUG::2014-09-25 17:14:48,178::task::1191::Storage.TaskManager.Task::(prepare) Task=`317b1722-f8ee-4daa-9939-52eb03a0daa2`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000431941', 'lastCheck': '6.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000456287', 'lastCheck': '5.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00294033', 'lastCheck': '1.1', 'valid': True}} >Thread-69896::DEBUG::2014-09-25 17:14:48,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`317b1722-f8ee-4daa-9939-52eb03a0daa2`::moving from state preparing -> state finished >Thread-69896::DEBUG::2014-09-25 17:14:48,179::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69896::DEBUG::2014-09-25 17:14:48,179::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69896::DEBUG::2014-09-25 17:14:48,179::task::993::Storage.TaskManager.Task::(_decref) Task=`317b1722-f8ee-4daa-9939-52eb03a0daa2`::ref 0 aborting False >Thread-69896::DEBUG::2014-09-25 17:14:48,181::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:48,203::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:48,204::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69897::DEBUG::2014-09-25 17:14:48,206::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:48,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:48,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146179 s, 70.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:14:50,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:50,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154026 s, 66.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:51,289::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:51,290::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69898::DEBUG::2014-09-25 17:14:51,292::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:14:51,677::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:51,678::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:51,678::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:51,678::__init__::232::IOProcess::(_processLogs) (107525) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:51,684::__init__::232::IOProcess::(_processLogs) (107525) Queuing response >Thread-79::DEBUG::2014-09-25 17:14:51,684::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:14:51,700::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.004252 s, 132 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:51,702::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:51,702::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:51,702::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:51,703::__init__::232::IOProcess::(_processLogs) (107526) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:51,704::__init__::232::IOProcess::(_processLogs) (107526) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:51,705::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:51,705::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:51,706::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:51,706::__init__::232::IOProcess::(_processLogs) (107527) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:51,706::__init__::232::IOProcess::(_processLogs) (107527) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:51,707::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:51,707::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:51,708::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:51,708::__init__::232::IOProcess::(_processLogs) (107528) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:51,709::__init__::232::IOProcess::(_processLogs) (107528) Queuing response >Thread-29::DEBUG::2014-09-25 17:14:51,709::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:51,709::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:51,710::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:51,711::__init__::232::IOProcess::(_processLogs) (107529) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:14:51,711::__init__::232::IOProcess::(_processLogs) (107529) Queuing response >Dummy-148::DEBUG::2014-09-25 17:14:52,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:14:52,660::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:52,668::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:52,668::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:52,669::__init__::232::IOProcess::(_processLogs) (107530) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:52,669::__init__::232::IOProcess::(_processLogs) (107530) Queuing response >Thread-80::DEBUG::2014-09-25 17:14:52,671::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:14:52,679::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000406274 s, 930 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:52,680::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:52,680::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:52,681::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:52,681::__init__::232::IOProcess::(_processLogs) (107531) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:52,682::__init__::232::IOProcess::(_processLogs) (107531) Queuing response >Dummy-148::DEBUG::2014-09-25 17:14:52,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171303 s, 59.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:54,306::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:54,308::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69899::DEBUG::2014-09-25 17:14:54,310::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:54,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:54,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172051 s, 59.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:56,701::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69900::DEBUG::2014-09-25 17:14:56,702::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:14:56,702::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69900::DEBUG::2014-09-25 17:14:56,704::task::595::Storage.TaskManager.Task::(_updateState) Task=`ff3018fc-f02c-4cb3-b0d4-b18176e6faab`::moving from state init -> state preparing >Thread-69900::INFO::2014-09-25 17:14:56,705::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69900::INFO::2014-09-25 17:14:56,705::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69900::DEBUG::2014-09-25 17:14:56,706::task::1191::Storage.TaskManager.Task::(prepare) Task=`ff3018fc-f02c-4cb3-b0d4-b18176e6faab`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69900::DEBUG::2014-09-25 17:14:56,706::task::595::Storage.TaskManager.Task::(_updateState) Task=`ff3018fc-f02c-4cb3-b0d4-b18176e6faab`::moving from state preparing -> state finished >Thread-69900::DEBUG::2014-09-25 17:14:56,706::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69900::DEBUG::2014-09-25 17:14:56,706::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69900::DEBUG::2014-09-25 17:14:56,706::task::993::Storage.TaskManager.Task::(_decref) Task=`ff3018fc-f02c-4cb3-b0d4-b18176e6faab`::ref 0 aborting False >Thread-69900::DEBUG::2014-09-25 17:14:56,707::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69900::DEBUG::2014-09-25 17:14:56,707::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:56,721::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:56,722::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69901::DEBUG::2014-09-25 17:14:56,722::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69901::DEBUG::2014-09-25 17:14:56,724::task::595::Storage.TaskManager.Task::(_updateState) Task=`8c554e68-b1c5-4403-80d8-5a1efe73bb9a`::moving from state init -> state preparing >Thread-69901::INFO::2014-09-25 17:14:56,724::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69901::DEBUG::2014-09-25 17:14:56,725::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c74bee08-eb54-470e-99bb-be5cdfd81ae5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69901::DEBUG::2014-09-25 17:14:56,725::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69901::DEBUG::2014-09-25 17:14:56,726::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69901::DEBUG::2014-09-25 17:14:56,726::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c74bee08-eb54-470e-99bb-be5cdfd81ae5`::Granted request >Thread-69901::DEBUG::2014-09-25 17:14:56,726::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`8c554e68-b1c5-4403-80d8-5a1efe73bb9a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69901::DEBUG::2014-09-25 17:14:56,727::task::993::Storage.TaskManager.Task::(_decref) Task=`8c554e68-b1c5-4403-80d8-5a1efe73bb9a`::ref 1 aborting False >Thread-69901::INFO::2014-09-25 17:14:56,731::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69901::DEBUG::2014-09-25 17:14:56,731::task::1191::Storage.TaskManager.Task::(prepare) Task=`8c554e68-b1c5-4403-80d8-5a1efe73bb9a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69901::DEBUG::2014-09-25 17:14:56,732::task::595::Storage.TaskManager.Task::(_updateState) Task=`8c554e68-b1c5-4403-80d8-5a1efe73bb9a`::moving from state preparing -> state finished >Thread-69901::DEBUG::2014-09-25 17:14:56,732::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69901::DEBUG::2014-09-25 17:14:56,732::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69901::DEBUG::2014-09-25 17:14:56,732::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69901::DEBUG::2014-09-25 17:14:56,733::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69901::DEBUG::2014-09-25 17:14:56,733::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69901::DEBUG::2014-09-25 17:14:56,733::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69901::DEBUG::2014-09-25 17:14:56,734::task::993::Storage.TaskManager.Task::(_decref) Task=`8c554e68-b1c5-4403-80d8-5a1efe73bb9a`::ref 0 aborting False >Thread-69901::DEBUG::2014-09-25 17:14:56,734::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69901::DEBUG::2014-09-25 17:14:56,734::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:56,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:56,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210767 s, 48.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:57,112::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:57,113::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:57,113::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:57,113::__init__::232::IOProcess::(_processLogs) (107532) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:57,126::__init__::232::IOProcess::(_processLogs) (107532) Queuing response >Thread-81::DEBUG::2014-09-25 17:14:57,126::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:14:57,150::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00241855 s, 153 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:14:57,151::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:14:57,151::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:14:57,152::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:14:57,152::__init__::232::IOProcess::(_processLogs) (107533) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:14:57,155::__init__::232::IOProcess::(_processLogs) (107533) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:14:57,323::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:14:57,325::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69902::DEBUG::2014-09-25 17:14:57,327::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:14:58,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:14:58,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147342 s, 69.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:00,341::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:00,343::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69903::DEBUG::2014-09-25 17:15:00,345::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:00,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:00,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175593 s, 58.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:01,712::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:01,712::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:01,712::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:01,713::__init__::232::IOProcess::(_processLogs) (107534) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:01,714::__init__::232::IOProcess::(_processLogs) (107534) Queuing response >Thread-79::DEBUG::2014-09-25 17:15:01,716::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:15:01,734::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00349667 s, 161 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:01,735::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:01,736::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:01,736::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:01,736::__init__::232::IOProcess::(_processLogs) (107535) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:01,737::__init__::232::IOProcess::(_processLogs) (107535) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:01,738::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:01,739::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:01,740::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:01,740::__init__::232::IOProcess::(_processLogs) (107536) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:01,740::__init__::232::IOProcess::(_processLogs) (107536) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:01,741::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:01,741::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:01,741::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:01,742::__init__::232::IOProcess::(_processLogs) (107537) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:01,742::__init__::232::IOProcess::(_processLogs) (107537) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:01,743::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:01,743::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:01,744::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:01,745::__init__::232::IOProcess::(_processLogs) (107538) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:01,745::__init__::232::IOProcess::(_processLogs) (107538) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:02,685::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:02,686::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:02,688::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:02,688::__init__::232::IOProcess::(_processLogs) (107539) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:02,690::__init__::232::IOProcess::(_processLogs) (107539) Queuing response >Thread-80::DEBUG::2014-09-25 17:15:02,692::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:15:02,704::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00244921 s, 154 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:02,705::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:02,705::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:02,705::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:02,706::__init__::232::IOProcess::(_processLogs) (107540) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:02,706::__init__::232::IOProcess::(_processLogs) (107540) Queuing response >Dummy-148::DEBUG::2014-09-25 17:15:02,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:02,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212441 s, 48.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:03,367::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:03,368::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69904::DEBUG::2014-09-25 17:15:03,371::task::595::Storage.TaskManager.Task::(_updateState) Task=`d5d2616d-bba1-459e-bf77-12ea5b5ddb68`::moving from state init -> state preparing >Thread-69904::INFO::2014-09-25 17:15:03,371::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69904::INFO::2014-09-25 17:15:03,372::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00349667', 'lastCheck': '1.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00244921', 'lastCheck': '0.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00241855', 'lastCheck': '6.2', 'valid': True}} >Thread-69904::DEBUG::2014-09-25 17:15:03,372::task::1191::Storage.TaskManager.Task::(prepare) Task=`d5d2616d-bba1-459e-bf77-12ea5b5ddb68`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00349667', 'lastCheck': '1.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00244921', 'lastCheck': '0.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00241855', 'lastCheck': '6.2', 'valid': True}} >Thread-69904::DEBUG::2014-09-25 17:15:03,372::task::595::Storage.TaskManager.Task::(_updateState) Task=`d5d2616d-bba1-459e-bf77-12ea5b5ddb68`::moving from state preparing -> state finished >Thread-69904::DEBUG::2014-09-25 17:15:03,373::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69904::DEBUG::2014-09-25 17:15:03,373::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69904::DEBUG::2014-09-25 17:15:03,373::task::993::Storage.TaskManager.Task::(_decref) Task=`d5d2616d-bba1-459e-bf77-12ea5b5ddb68`::ref 0 aborting False >Thread-69904::DEBUG::2014-09-25 17:15:03,376::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:03,391::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:03,393::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69905::DEBUG::2014-09-25 17:15:03,395::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:04,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:04,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.02182 s, 46.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:06,445::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:06,446::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69906::DEBUG::2014-09-25 17:15:06,448::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:06,782::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:06,783::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69907::DEBUG::2014-09-25 17:15:06,783::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69907::DEBUG::2014-09-25 17:15:06,785::task::595::Storage.TaskManager.Task::(_updateState) Task=`07a9c579-139e-46d1-a2b2-b4f6352f5bef`::moving from state init -> state preparing >Thread-69907::INFO::2014-09-25 17:15:06,785::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69907::INFO::2014-09-25 17:15:06,785::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69907::DEBUG::2014-09-25 17:15:06,786::task::1191::Storage.TaskManager.Task::(prepare) Task=`07a9c579-139e-46d1-a2b2-b4f6352f5bef`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69907::DEBUG::2014-09-25 17:15:06,786::task::595::Storage.TaskManager.Task::(_updateState) Task=`07a9c579-139e-46d1-a2b2-b4f6352f5bef`::moving from state preparing -> state finished >Thread-69907::DEBUG::2014-09-25 17:15:06,786::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69907::DEBUG::2014-09-25 17:15:06,786::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69907::DEBUG::2014-09-25 17:15:06,787::task::993::Storage.TaskManager.Task::(_decref) Task=`07a9c579-139e-46d1-a2b2-b4f6352f5bef`::ref 0 aborting False >Thread-69907::DEBUG::2014-09-25 17:15:06,787::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69907::DEBUG::2014-09-25 17:15:06,787::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:06,801::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:06,802::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69908::DEBUG::2014-09-25 17:15:06,802::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69908::DEBUG::2014-09-25 17:15:06,804::task::595::Storage.TaskManager.Task::(_updateState) Task=`6b4b3b75-b964-46e9-9390-14fececa2f94`::moving from state init -> state preparing >Thread-69908::INFO::2014-09-25 17:15:06,804::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69908::DEBUG::2014-09-25 17:15:06,805::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`87f7dd9e-3499-4b89-b20c-0e31e48eb319`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69908::DEBUG::2014-09-25 17:15:06,805::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69908::DEBUG::2014-09-25 17:15:06,806::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69908::DEBUG::2014-09-25 17:15:06,806::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`87f7dd9e-3499-4b89-b20c-0e31e48eb319`::Granted request >Thread-69908::DEBUG::2014-09-25 17:15:06,806::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`6b4b3b75-b964-46e9-9390-14fececa2f94`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69908::DEBUG::2014-09-25 17:15:06,807::task::993::Storage.TaskManager.Task::(_decref) Task=`6b4b3b75-b964-46e9-9390-14fececa2f94`::ref 1 aborting False >Thread-69908::INFO::2014-09-25 17:15:06,811::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69908::DEBUG::2014-09-25 17:15:06,811::task::1191::Storage.TaskManager.Task::(prepare) Task=`6b4b3b75-b964-46e9-9390-14fececa2f94`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69908::DEBUG::2014-09-25 17:15:06,811::task::595::Storage.TaskManager.Task::(_updateState) Task=`6b4b3b75-b964-46e9-9390-14fececa2f94`::moving from state preparing -> state finished >Thread-69908::DEBUG::2014-09-25 17:15:06,812::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69908::DEBUG::2014-09-25 17:15:06,812::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69908::DEBUG::2014-09-25 17:15:06,812::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69908::DEBUG::2014-09-25 17:15:06,813::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69908::DEBUG::2014-09-25 17:15:06,813::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69908::DEBUG::2014-09-25 17:15:06,813::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69908::DEBUG::2014-09-25 17:15:06,813::task::993::Storage.TaskManager.Task::(_decref) Task=`6b4b3b75-b964-46e9-9390-14fececa2f94`::ref 0 aborting False >Thread-69908::DEBUG::2014-09-25 17:15:06,814::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69908::DEBUG::2014-09-25 17:15:06,814::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:07,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:07,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183642 s, 55.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:07,157::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:07,157::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:07,157::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:07,158::__init__::232::IOProcess::(_processLogs) (107541) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:07,160::__init__::232::IOProcess::(_processLogs) (107541) Queuing response >Thread-81::DEBUG::2014-09-25 17:15:07,161::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:15:07,186::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00843838 s, 44.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:07,187::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:07,187::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:07,188::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:07,188::__init__::232::IOProcess::(_processLogs) (107542) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:07,193::__init__::232::IOProcess::(_processLogs) (107542) Queuing response >Dummy-148::DEBUG::2014-09-25 17:15:09,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:09,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153564 s, 66.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:09,464::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:09,465::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69909::DEBUG::2014-09-25 17:15:09,468::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:11,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:11,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159133 s, 64.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:11,745::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:11,745::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:11,746::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:11,746::__init__::232::IOProcess::(_processLogs) (107543) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:11,756::__init__::232::IOProcess::(_processLogs) (107543) Queuing response >Thread-79::DEBUG::2014-09-25 17:15:11,757::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:15:11,778::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00488648 s, 115 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:11,779::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:11,779::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:11,780::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:11,780::__init__::232::IOProcess::(_processLogs) (107544) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:11,782::__init__::232::IOProcess::(_processLogs) (107544) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:11,783::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:11,784::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:11,785::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:11,785::__init__::232::IOProcess::(_processLogs) (107545) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:11,785::__init__::232::IOProcess::(_processLogs) (107545) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:11,786::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:11,786::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:11,787::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:11,787::__init__::232::IOProcess::(_processLogs) (107546) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:11,787::__init__::232::IOProcess::(_processLogs) (107546) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:11,788::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:11,788::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:11,789::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:11,789::__init__::232::IOProcess::(_processLogs) (107547) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:11,789::__init__::232::IOProcess::(_processLogs) (107547) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:12,485::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:12,486::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69910::DEBUG::2014-09-25 17:15:12,488::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:15:12,707::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:12,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:12,708::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:12,708::__init__::232::IOProcess::(_processLogs) (107548) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:12,709::__init__::232::IOProcess::(_processLogs) (107548) Queuing response >Thread-80::DEBUG::2014-09-25 17:15:12,710::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:15:12,719::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.0005634 s, 671 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:12,720::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:12,720::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:12,720::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:12,721::__init__::232::IOProcess::(_processLogs) (107549) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:12,721::__init__::232::IOProcess::(_processLogs) (107549) Queuing response >Dummy-148::DEBUG::2014-09-25 17:15:13,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:13,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165028 s, 62.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:15:15,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:15,504::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:15,506::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69911::DEBUG::2014-09-25 17:15:15,508::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:15,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0445836 s, 23.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:16,841::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69912::DEBUG::2014-09-25 17:15:16,844::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:15:16,844::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69912::DEBUG::2014-09-25 17:15:16,849::task::595::Storage.TaskManager.Task::(_updateState) Task=`bc8e435a-d164-4976-9325-113fa760177e`::moving from state init -> state preparing >Thread-69912::INFO::2014-09-25 17:15:16,850::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69912::INFO::2014-09-25 17:15:16,852::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69912::DEBUG::2014-09-25 17:15:16,854::task::1191::Storage.TaskManager.Task::(prepare) Task=`bc8e435a-d164-4976-9325-113fa760177e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69912::DEBUG::2014-09-25 17:15:16,855::task::595::Storage.TaskManager.Task::(_updateState) Task=`bc8e435a-d164-4976-9325-113fa760177e`::moving from state preparing -> state finished >Thread-69912::DEBUG::2014-09-25 17:15:16,855::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69912::DEBUG::2014-09-25 17:15:16,856::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69912::DEBUG::2014-09-25 17:15:16,858::task::993::Storage.TaskManager.Task::(_decref) Task=`bc8e435a-d164-4976-9325-113fa760177e`::ref 0 aborting False >Thread-69912::DEBUG::2014-09-25 17:15:16,859::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69912::DEBUG::2014-09-25 17:15:16,860::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:16,876::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:16,879::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69913::DEBUG::2014-09-25 17:15:16,879::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69913::DEBUG::2014-09-25 17:15:16,884::task::595::Storage.TaskManager.Task::(_updateState) Task=`59024c31-dd47-4686-ae01-32e501f18205`::moving from state init -> state preparing >Thread-69913::INFO::2014-09-25 17:15:16,885::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69913::DEBUG::2014-09-25 17:15:16,888::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c30f518e-3966-4a25-ab19-f6ef85655072`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69913::DEBUG::2014-09-25 17:15:16,889::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69913::DEBUG::2014-09-25 17:15:16,891::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69913::DEBUG::2014-09-25 17:15:16,892::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c30f518e-3966-4a25-ab19-f6ef85655072`::Granted request >Thread-69913::DEBUG::2014-09-25 17:15:16,895::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`59024c31-dd47-4686-ae01-32e501f18205`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69913::DEBUG::2014-09-25 17:15:16,896::task::993::Storage.TaskManager.Task::(_decref) Task=`59024c31-dd47-4686-ae01-32e501f18205`::ref 1 aborting False >Thread-69913::INFO::2014-09-25 17:15:16,916::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69913::DEBUG::2014-09-25 17:15:16,917::task::1191::Storage.TaskManager.Task::(prepare) Task=`59024c31-dd47-4686-ae01-32e501f18205`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69913::DEBUG::2014-09-25 17:15:16,918::task::595::Storage.TaskManager.Task::(_updateState) Task=`59024c31-dd47-4686-ae01-32e501f18205`::moving from state preparing -> state finished >Thread-69913::DEBUG::2014-09-25 17:15:16,919::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69913::DEBUG::2014-09-25 17:15:16,920::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69913::DEBUG::2014-09-25 17:15:16,921::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69913::DEBUG::2014-09-25 17:15:16,923::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69913::DEBUG::2014-09-25 17:15:16,924::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69913::DEBUG::2014-09-25 17:15:16,925::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69913::DEBUG::2014-09-25 17:15:16,927::task::993::Storage.TaskManager.Task::(_decref) Task=`59024c31-dd47-4686-ae01-32e501f18205`::ref 0 aborting False >Thread-69913::DEBUG::2014-09-25 17:15:16,928::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69913::DEBUG::2014-09-25 17:15:16,930::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:15:17,195::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:17,195::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:17,195::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:17,197::__init__::232::IOProcess::(_processLogs) (107550) Got request for method 'statvfs' >Thread-81::DEBUG::2014-09-25 17:15:17,197::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:15:17,198::__init__::232::IOProcess::(_processLogs) (107550) Queuing response >Thread-81::DEBUG::2014-09-25 17:15:17,207::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000438878 s, 845 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:17,208::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:17,208::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:17,208::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:17,208::__init__::232::IOProcess::(_processLogs) (107551) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:17,209::__init__::232::IOProcess::(_processLogs) (107551) Queuing response >Dummy-148::DEBUG::2014-09-25 17:15:17,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:17,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204166 s, 50.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:18,520::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:18,522::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69914::DEBUG::2014-09-25 17:15:18,526::task::595::Storage.TaskManager.Task::(_updateState) Task=`c5348772-204a-4efb-a9ce-97e836bdad1e`::moving from state init -> state preparing >Thread-69914::INFO::2014-09-25 17:15:18,526::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69914::INFO::2014-09-25 17:15:18,527::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00488648', 'lastCheck': '6.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0005634', 'lastCheck': '5.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000438878', 'lastCheck': '1.3', 'valid': True}} >Thread-69914::DEBUG::2014-09-25 17:15:18,527::task::1191::Storage.TaskManager.Task::(prepare) Task=`c5348772-204a-4efb-a9ce-97e836bdad1e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00488648', 'lastCheck': '6.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0005634', 'lastCheck': '5.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000438878', 'lastCheck': '1.3', 'valid': True}} >Thread-69914::DEBUG::2014-09-25 17:15:18,527::task::595::Storage.TaskManager.Task::(_updateState) Task=`c5348772-204a-4efb-a9ce-97e836bdad1e`::moving from state preparing -> state finished >Thread-69914::DEBUG::2014-09-25 17:15:18,527::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69914::DEBUG::2014-09-25 17:15:18,528::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69914::DEBUG::2014-09-25 17:15:18,528::task::993::Storage.TaskManager.Task::(_decref) Task=`c5348772-204a-4efb-a9ce-97e836bdad1e`::ref 0 aborting False >Thread-69914::DEBUG::2014-09-25 17:15:18,530::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:18,551::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:18,552::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69915::DEBUG::2014-09-25 17:15:18,554::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:19,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:19,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136664 s, 74.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:21,601::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:21,603::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69916::DEBUG::2014-09-25 17:15:21,605::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:15:21,790::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:21,790::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:21,790::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:21,791::__init__::232::IOProcess::(_processLogs) (107552) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:21,793::__init__::232::IOProcess::(_processLogs) (107552) Queuing response >Thread-79::DEBUG::2014-09-25 17:15:21,793::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:15:21,807::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00476981 s, 118 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:21,809::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:21,809::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:21,809::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:21,809::__init__::232::IOProcess::(_processLogs) (107553) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:21,812::__init__::232::IOProcess::(_processLogs) (107553) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:21,813::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:21,814::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:21,814::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:21,815::__init__::232::IOProcess::(_processLogs) (107554) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:21,815::__init__::232::IOProcess::(_processLogs) (107554) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:21,816::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:21,816::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:21,816::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:21,817::__init__::232::IOProcess::(_processLogs) (107555) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:21,817::__init__::232::IOProcess::(_processLogs) (107555) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:21,818::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:21,818::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:21,818::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:21,819::__init__::232::IOProcess::(_processLogs) (107556) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:21,820::__init__::232::IOProcess::(_processLogs) (107556) Queuing response >Dummy-148::DEBUG::2014-09-25 17:15:21,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:21,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171884 s, 59.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:22,724::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:22,724::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:22,724::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:22,724::__init__::232::IOProcess::(_processLogs) (107557) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:22,725::__init__::232::IOProcess::(_processLogs) (107557) Queuing response >Thread-80::DEBUG::2014-09-25 17:15:22,726::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:15:22,735::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000604678 s, 625 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:22,737::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:22,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:22,737::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:22,737::__init__::232::IOProcess::(_processLogs) (107558) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:22,738::__init__::232::IOProcess::(_processLogs) (107558) Queuing response >Dummy-148::DEBUG::2014-09-25 17:15:23,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:23,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167836 s, 61.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:24,620::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:24,621::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69917::DEBUG::2014-09-25 17:15:24,623::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:25,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:26,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175784 s, 58.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:26,961::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69918::DEBUG::2014-09-25 17:15:26,962::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:15:26,962::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69918::DEBUG::2014-09-25 17:15:26,964::task::595::Storage.TaskManager.Task::(_updateState) Task=`6419d929-1663-417d-96c9-32530204c2e5`::moving from state init -> state preparing >Thread-69918::INFO::2014-09-25 17:15:26,965::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69918::INFO::2014-09-25 17:15:26,965::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69918::DEBUG::2014-09-25 17:15:26,965::task::1191::Storage.TaskManager.Task::(prepare) Task=`6419d929-1663-417d-96c9-32530204c2e5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69918::DEBUG::2014-09-25 17:15:26,966::task::595::Storage.TaskManager.Task::(_updateState) Task=`6419d929-1663-417d-96c9-32530204c2e5`::moving from state preparing -> state finished >Thread-69918::DEBUG::2014-09-25 17:15:26,966::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69918::DEBUG::2014-09-25 17:15:26,966::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69918::DEBUG::2014-09-25 17:15:26,966::task::993::Storage.TaskManager.Task::(_decref) Task=`6419d929-1663-417d-96c9-32530204c2e5`::ref 0 aborting False >Thread-69918::DEBUG::2014-09-25 17:15:26,967::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69918::DEBUG::2014-09-25 17:15:26,967::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:26,983::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69919::DEBUG::2014-09-25 17:15:26,984::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:15:26,984::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69919::DEBUG::2014-09-25 17:15:26,986::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c691bb3-1a0c-440b-b662-cea7c62802d1`::moving from state init -> state preparing >Thread-69919::INFO::2014-09-25 17:15:26,986::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69919::DEBUG::2014-09-25 17:15:26,987::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c5e001a7-0adb-46b2-babd-e362c7eb440a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69919::DEBUG::2014-09-25 17:15:26,987::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69919::DEBUG::2014-09-25 17:15:26,988::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69919::DEBUG::2014-09-25 17:15:26,988::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c5e001a7-0adb-46b2-babd-e362c7eb440a`::Granted request >Thread-69919::DEBUG::2014-09-25 17:15:26,989::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2c691bb3-1a0c-440b-b662-cea7c62802d1`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69919::DEBUG::2014-09-25 17:15:26,989::task::993::Storage.TaskManager.Task::(_decref) Task=`2c691bb3-1a0c-440b-b662-cea7c62802d1`::ref 1 aborting False >Thread-69919::INFO::2014-09-25 17:15:26,993::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69919::DEBUG::2014-09-25 17:15:26,993::task::1191::Storage.TaskManager.Task::(prepare) Task=`2c691bb3-1a0c-440b-b662-cea7c62802d1`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69919::DEBUG::2014-09-25 17:15:26,994::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c691bb3-1a0c-440b-b662-cea7c62802d1`::moving from state preparing -> state finished >Thread-69919::DEBUG::2014-09-25 17:15:26,994::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69919::DEBUG::2014-09-25 17:15:26,994::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69919::DEBUG::2014-09-25 17:15:26,994::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69919::DEBUG::2014-09-25 17:15:26,995::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69919::DEBUG::2014-09-25 17:15:26,995::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69919::DEBUG::2014-09-25 17:15:26,995::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69919::DEBUG::2014-09-25 17:15:26,996::task::993::Storage.TaskManager.Task::(_decref) Task=`2c691bb3-1a0c-440b-b662-cea7c62802d1`::ref 0 aborting False >Thread-69919::DEBUG::2014-09-25 17:15:26,996::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69919::DEBUG::2014-09-25 17:15:26,997::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:15:27,211::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:27,211::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:27,212::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:27,212::__init__::232::IOProcess::(_processLogs) (107559) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:27,215::__init__::232::IOProcess::(_processLogs) (107559) Queuing response >Thread-81::DEBUG::2014-09-25 17:15:27,216::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:15:27,232::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0056672 s, 65.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:27,233::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:27,234::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:27,234::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:27,234::__init__::232::IOProcess::(_processLogs) (107560) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:27,236::__init__::232::IOProcess::(_processLogs) (107560) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:27,637::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:27,639::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69920::DEBUG::2014-09-25 17:15:27,640::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:28,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:28,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133085 s, 76.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:15:30,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:30,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157337 s, 65.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:30,656::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:30,657::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69921::DEBUG::2014-09-25 17:15:30,659::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:15:31,820::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:31,820::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:31,821::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:31,821::__init__::232::IOProcess::(_processLogs) (107561) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:31,828::__init__::232::IOProcess::(_processLogs) (107561) Queuing response >Thread-79::DEBUG::2014-09-25 17:15:31,829::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:15:31,849::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00849152 s, 66.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:31,850::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:31,851::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:31,851::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:31,851::__init__::232::IOProcess::(_processLogs) (107562) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:31,858::__init__::232::IOProcess::(_processLogs) (107562) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:31,860::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:31,860::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:31,861::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:31,861::__init__::232::IOProcess::(_processLogs) (107563) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:31,861::__init__::232::IOProcess::(_processLogs) (107563) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:31,862::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:31,862::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:31,863::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:31,864::__init__::232::IOProcess::(_processLogs) (107564) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:31,864::__init__::232::IOProcess::(_processLogs) (107564) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:31,864::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:31,865::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:31,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:31,866::__init__::232::IOProcess::(_processLogs) (107565) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:31,866::__init__::232::IOProcess::(_processLogs) (107565) Queuing response >Dummy-148::DEBUG::2014-09-25 17:15:32,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:32,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205497 s, 49.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:32,741::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:32,741::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:32,741::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:32,742::__init__::232::IOProcess::(_processLogs) (107566) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:32,743::__init__::232::IOProcess::(_processLogs) (107566) Queuing response >Thread-80::DEBUG::2014-09-25 17:15:32,744::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:15:32,753::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000457106 s, 827 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:32,754::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:32,755::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:32,755::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:32,755::__init__::232::IOProcess::(_processLogs) (107567) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:32,756::__init__::232::IOProcess::(_processLogs) (107567) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:33,676::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:33,677::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69922::DEBUG::2014-09-25 17:15:33,680::task::595::Storage.TaskManager.Task::(_updateState) Task=`186712e4-4eeb-41b1-82f9-a905104f9089`::moving from state init -> state preparing >Thread-69922::INFO::2014-09-25 17:15:33,681::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69922::INFO::2014-09-25 17:15:33,681::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00849152', 'lastCheck': '1.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000457106', 'lastCheck': '0.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0056672', 'lastCheck': '6.4', 'valid': True}} >Thread-69922::DEBUG::2014-09-25 17:15:33,681::task::1191::Storage.TaskManager.Task::(prepare) Task=`186712e4-4eeb-41b1-82f9-a905104f9089`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00849152', 'lastCheck': '1.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000457106', 'lastCheck': '0.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0056672', 'lastCheck': '6.4', 'valid': True}} >Thread-69922::DEBUG::2014-09-25 17:15:33,682::task::595::Storage.TaskManager.Task::(_updateState) Task=`186712e4-4eeb-41b1-82f9-a905104f9089`::moving from state preparing -> state finished >Thread-69922::DEBUG::2014-09-25 17:15:33,682::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69922::DEBUG::2014-09-25 17:15:33,682::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69922::DEBUG::2014-09-25 17:15:33,682::task::993::Storage.TaskManager.Task::(_decref) Task=`186712e4-4eeb-41b1-82f9-a905104f9089`::ref 0 aborting False >Thread-69922::DEBUG::2014-09-25 17:15:33,685::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:33,705::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:33,706::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69923::DEBUG::2014-09-25 17:15:33,708::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:34,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:34,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0198186 s, 51.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:15:36,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:36,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144318 s, 71.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:36,779::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:36,780::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69924::DEBUG::2014-09-25 17:15:36,782::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:37,027::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69925::DEBUG::2014-09-25 17:15:37,029::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:15:37,029::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69925::DEBUG::2014-09-25 17:15:37,031::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb39eba3-8d2a-42c8-8fd8-bf2f6b47763b`::moving from state init -> state preparing >Thread-69925::INFO::2014-09-25 17:15:37,031::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69925::INFO::2014-09-25 17:15:37,032::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69925::DEBUG::2014-09-25 17:15:37,032::task::1191::Storage.TaskManager.Task::(prepare) Task=`fb39eba3-8d2a-42c8-8fd8-bf2f6b47763b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69925::DEBUG::2014-09-25 17:15:37,032::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb39eba3-8d2a-42c8-8fd8-bf2f6b47763b`::moving from state preparing -> state finished >Thread-69925::DEBUG::2014-09-25 17:15:37,033::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69925::DEBUG::2014-09-25 17:15:37,033::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69925::DEBUG::2014-09-25 17:15:37,033::task::993::Storage.TaskManager.Task::(_decref) Task=`fb39eba3-8d2a-42c8-8fd8-bf2f6b47763b`::ref 0 aborting False >Thread-69925::DEBUG::2014-09-25 17:15:37,033::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69925::DEBUG::2014-09-25 17:15:37,034::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:37,044::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69926::DEBUG::2014-09-25 17:15:37,045::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:15:37,045::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69926::DEBUG::2014-09-25 17:15:37,048::task::595::Storage.TaskManager.Task::(_updateState) Task=`fd5cabf6-2f9e-4f7d-889d-38e9ff9105f1`::moving from state init -> state preparing >Thread-69926::INFO::2014-09-25 17:15:37,048::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69926::DEBUG::2014-09-25 17:15:37,049::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cc7ddc4d-3e2e-4fc2-a8e0-12c3d9d50a5e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69926::DEBUG::2014-09-25 17:15:37,049::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69926::DEBUG::2014-09-25 17:15:37,049::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69926::DEBUG::2014-09-25 17:15:37,050::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cc7ddc4d-3e2e-4fc2-a8e0-12c3d9d50a5e`::Granted request >Thread-69926::DEBUG::2014-09-25 17:15:37,050::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`fd5cabf6-2f9e-4f7d-889d-38e9ff9105f1`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69926::DEBUG::2014-09-25 17:15:37,050::task::993::Storage.TaskManager.Task::(_decref) Task=`fd5cabf6-2f9e-4f7d-889d-38e9ff9105f1`::ref 1 aborting False >Thread-69926::INFO::2014-09-25 17:15:37,055::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69926::DEBUG::2014-09-25 17:15:37,055::task::1191::Storage.TaskManager.Task::(prepare) Task=`fd5cabf6-2f9e-4f7d-889d-38e9ff9105f1`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69926::DEBUG::2014-09-25 17:15:37,055::task::595::Storage.TaskManager.Task::(_updateState) Task=`fd5cabf6-2f9e-4f7d-889d-38e9ff9105f1`::moving from state preparing -> state finished >Thread-69926::DEBUG::2014-09-25 17:15:37,056::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69926::DEBUG::2014-09-25 17:15:37,056::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69926::DEBUG::2014-09-25 17:15:37,056::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69926::DEBUG::2014-09-25 17:15:37,056::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69926::DEBUG::2014-09-25 17:15:37,057::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69926::DEBUG::2014-09-25 17:15:37,057::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69926::DEBUG::2014-09-25 17:15:37,057::task::993::Storage.TaskManager.Task::(_decref) Task=`fd5cabf6-2f9e-4f7d-889d-38e9ff9105f1`::ref 0 aborting False >Thread-69926::DEBUG::2014-09-25 17:15:37,058::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69926::DEBUG::2014-09-25 17:15:37,058::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:15:37,238::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:37,238::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-81::DEBUG::2014-09-25 17:15:37,240::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:15:37,240::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:37,248::__init__::232::IOProcess::(_processLogs) (107568) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:37,249::__init__::232::IOProcess::(_processLogs) (107568) Queuing response >Thread-81::DEBUG::2014-09-25 17:15:37,249::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000921897 s, 402 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:37,250::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:37,251::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:37,251::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:37,251::__init__::232::IOProcess::(_processLogs) (107569) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:37,251::__init__::232::IOProcess::(_processLogs) (107569) Queuing response >Dummy-148::DEBUG::2014-09-25 17:15:38,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:38,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179021 s, 57.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:39,797::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:39,798::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69927::DEBUG::2014-09-25 17:15:39,800::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:40,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:40,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016501 s, 62.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:41,867::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:41,867::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:41,867::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:41,868::__init__::232::IOProcess::(_processLogs) (107570) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:41,871::__init__::232::IOProcess::(_processLogs) (107570) Queuing response >Thread-79::DEBUG::2014-09-25 17:15:41,872::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:15:41,885::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00292307 s, 192 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:41,886::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:41,886::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:41,886::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:41,887::__init__::232::IOProcess::(_processLogs) (107571) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:41,889::__init__::232::IOProcess::(_processLogs) (107571) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:41,891::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:41,892::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:41,892::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:41,892::__init__::232::IOProcess::(_processLogs) (107572) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:41,892::__init__::232::IOProcess::(_processLogs) (107572) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:41,894::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:41,894::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:41,894::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:41,894::__init__::232::IOProcess::(_processLogs) (107573) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:41,898::__init__::232::IOProcess::(_processLogs) (107573) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:41,898::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:41,899::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:41,899::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:41,899::__init__::232::IOProcess::(_processLogs) (107574) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:41,903::__init__::232::IOProcess::(_processLogs) (107574) Queuing response >Dummy-148::DEBUG::2014-09-25 17:15:42,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:42,399::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133837 s, 76.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:42,758::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:42,758::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:42,758::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:42,759::__init__::232::IOProcess::(_processLogs) (107575) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:42,759::__init__::232::IOProcess::(_processLogs) (107575) Queuing response >Thread-80::DEBUG::2014-09-25 17:15:42,761::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:15:42,770::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000529269 s, 714 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:42,771::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:42,771::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:42,772::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:42,774::__init__::232::IOProcess::(_processLogs) (107576) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:42,774::__init__::232::IOProcess::(_processLogs) (107576) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:42,815::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:42,816::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69928::DEBUG::2014-09-25 17:15:42,818::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:44,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:44,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.236635 s, 4.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:45,835::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:45,837::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69929::DEBUG::2014-09-25 17:15:45,839::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:46,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:46,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0231411 s, 44.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:47,081::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69930::DEBUG::2014-09-25 17:15:47,082::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:15:47,082::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69930::DEBUG::2014-09-25 17:15:47,085::task::595::Storage.TaskManager.Task::(_updateState) Task=`286dc41d-b2fe-412e-b663-4e07f09c4d75`::moving from state init -> state preparing >Thread-69930::INFO::2014-09-25 17:15:47,085::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69930::INFO::2014-09-25 17:15:47,085::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69930::DEBUG::2014-09-25 17:15:47,086::task::1191::Storage.TaskManager.Task::(prepare) Task=`286dc41d-b2fe-412e-b663-4e07f09c4d75`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69930::DEBUG::2014-09-25 17:15:47,086::task::595::Storage.TaskManager.Task::(_updateState) Task=`286dc41d-b2fe-412e-b663-4e07f09c4d75`::moving from state preparing -> state finished >Thread-69930::DEBUG::2014-09-25 17:15:47,086::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69930::DEBUG::2014-09-25 17:15:47,086::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69930::DEBUG::2014-09-25 17:15:47,086::task::993::Storage.TaskManager.Task::(_decref) Task=`286dc41d-b2fe-412e-b663-4e07f09c4d75`::ref 0 aborting False >Thread-69930::DEBUG::2014-09-25 17:15:47,087::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69930::DEBUG::2014-09-25 17:15:47,087::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:47,097::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69931::DEBUG::2014-09-25 17:15:47,098::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:15:47,098::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69931::DEBUG::2014-09-25 17:15:47,100::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a5a7ddf-c215-4739-bbfc-15acbb6cae3e`::moving from state init -> state preparing >Thread-69931::INFO::2014-09-25 17:15:47,100::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69931::DEBUG::2014-09-25 17:15:47,101::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1d133d42-9a46-414e-a41d-461f263845bf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69931::DEBUG::2014-09-25 17:15:47,101::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69931::DEBUG::2014-09-25 17:15:47,101::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69931::DEBUG::2014-09-25 17:15:47,102::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1d133d42-9a46-414e-a41d-461f263845bf`::Granted request >Thread-69931::DEBUG::2014-09-25 17:15:47,102::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9a5a7ddf-c215-4739-bbfc-15acbb6cae3e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69931::DEBUG::2014-09-25 17:15:47,102::task::993::Storage.TaskManager.Task::(_decref) Task=`9a5a7ddf-c215-4739-bbfc-15acbb6cae3e`::ref 1 aborting False >Thread-69931::INFO::2014-09-25 17:15:47,106::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69931::DEBUG::2014-09-25 17:15:47,107::task::1191::Storage.TaskManager.Task::(prepare) Task=`9a5a7ddf-c215-4739-bbfc-15acbb6cae3e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69931::DEBUG::2014-09-25 17:15:47,107::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a5a7ddf-c215-4739-bbfc-15acbb6cae3e`::moving from state preparing -> state finished >Thread-69931::DEBUG::2014-09-25 17:15:47,107::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69931::DEBUG::2014-09-25 17:15:47,107::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69931::DEBUG::2014-09-25 17:15:47,108::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69931::DEBUG::2014-09-25 17:15:47,108::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69931::DEBUG::2014-09-25 17:15:47,108::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69931::DEBUG::2014-09-25 17:15:47,108::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69931::DEBUG::2014-09-25 17:15:47,109::task::993::Storage.TaskManager.Task::(_decref) Task=`9a5a7ddf-c215-4739-bbfc-15acbb6cae3e`::ref 0 aborting False >Thread-69931::DEBUG::2014-09-25 17:15:47,109::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69931::DEBUG::2014-09-25 17:15:47,109::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:15:47,253::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:47,253::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:47,253::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:47,254::__init__::232::IOProcess::(_processLogs) (107577) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:47,256::__init__::232::IOProcess::(_processLogs) (107577) Queuing response >Thread-81::DEBUG::2014-09-25 17:15:47,258::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:15:47,274::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00379952 s, 97.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:47,275::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:47,275::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:47,275::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:47,276::__init__::232::IOProcess::(_processLogs) (107578) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:47,278::__init__::232::IOProcess::(_processLogs) (107578) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:48,854::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:48,856::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69932::DEBUG::2014-09-25 17:15:48,858::task::595::Storage.TaskManager.Task::(_updateState) Task=`11b99891-68b7-421d-8e51-3126d2a1a634`::moving from state init -> state preparing >Thread-69932::INFO::2014-09-25 17:15:48,859::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69932::INFO::2014-09-25 17:15:48,859::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00292307', 'lastCheck': '7.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000529269', 'lastCheck': '6.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00379952', 'lastCheck': '1.6', 'valid': True}} >Thread-69932::DEBUG::2014-09-25 17:15:48,859::task::1191::Storage.TaskManager.Task::(prepare) Task=`11b99891-68b7-421d-8e51-3126d2a1a634`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00292307', 'lastCheck': '7.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000529269', 'lastCheck': '6.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00379952', 'lastCheck': '1.6', 'valid': True}} >Thread-69932::DEBUG::2014-09-25 17:15:48,860::task::595::Storage.TaskManager.Task::(_updateState) Task=`11b99891-68b7-421d-8e51-3126d2a1a634`::moving from state preparing -> state finished >Thread-69932::DEBUG::2014-09-25 17:15:48,860::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69932::DEBUG::2014-09-25 17:15:48,860::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69932::DEBUG::2014-09-25 17:15:48,860::task::993::Storage.TaskManager.Task::(_decref) Task=`11b99891-68b7-421d-8e51-3126d2a1a634`::ref 0 aborting False >Thread-69932::DEBUG::2014-09-25 17:15:48,863::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:48,894::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:48,896::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69933::DEBUG::2014-09-25 17:15:48,901::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:48,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:48,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0260808 s, 39.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:15:50,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:51,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161812 s, 63.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:51,905::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:51,905::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:51,905::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:51,906::__init__::232::IOProcess::(_processLogs) (107579) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:51,906::__init__::232::IOProcess::(_processLogs) (107579) Queuing response >Thread-79::DEBUG::2014-09-25 17:15:51,908::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:15:51,917::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000387993 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:51,918::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:51,919::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:51,919::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:51,919::__init__::232::IOProcess::(_processLogs) (107580) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:51,920::__init__::232::IOProcess::(_processLogs) (107580) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:51,921::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:51,921::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:51,922::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:51,922::__init__::232::IOProcess::(_processLogs) (107581) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:51,923::__init__::232::IOProcess::(_processLogs) (107581) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:51,923::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:51,924::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:51,924::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:51,924::__init__::232::IOProcess::(_processLogs) (107582) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:51,925::__init__::232::IOProcess::(_processLogs) (107582) Queuing response >Thread-29::DEBUG::2014-09-25 17:15:51,925::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:51,926::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:51,927::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:51,927::__init__::232::IOProcess::(_processLogs) (107583) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:15:51,927::__init__::232::IOProcess::(_processLogs) (107583) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:51,964::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:51,965::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69934::DEBUG::2014-09-25 17:15:51,967::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:15:52,774::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:52,775::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:52,775::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:52,775::__init__::232::IOProcess::(_processLogs) (107584) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:52,776::__init__::232::IOProcess::(_processLogs) (107584) Queuing response >Thread-80::DEBUG::2014-09-25 17:15:52,777::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:15:52,786::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000447803 s, 844 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:52,787::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:52,787::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:52,787::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:52,788::__init__::232::IOProcess::(_processLogs) (107585) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:52,788::__init__::232::IOProcess::(_processLogs) (107585) Queuing response >Dummy-148::DEBUG::2014-09-25 17:15:53,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:53,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192722 s, 53.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:54,986::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:54,987::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69935::DEBUG::2014-09-25 17:15:54,989::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:55,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:55,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207639 s, 49.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:57,134::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69936::DEBUG::2014-09-25 17:15:57,136::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:15:57,136::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69936::DEBUG::2014-09-25 17:15:57,138::task::595::Storage.TaskManager.Task::(_updateState) Task=`c3cde9e8-7c49-435e-8861-d598e3b16ab4`::moving from state init -> state preparing >Thread-69936::INFO::2014-09-25 17:15:57,139::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69936::INFO::2014-09-25 17:15:57,139::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69936::DEBUG::2014-09-25 17:15:57,140::task::1191::Storage.TaskManager.Task::(prepare) Task=`c3cde9e8-7c49-435e-8861-d598e3b16ab4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69936::DEBUG::2014-09-25 17:15:57,140::task::595::Storage.TaskManager.Task::(_updateState) Task=`c3cde9e8-7c49-435e-8861-d598e3b16ab4`::moving from state preparing -> state finished >Thread-69936::DEBUG::2014-09-25 17:15:57,140::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69936::DEBUG::2014-09-25 17:15:57,140::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69936::DEBUG::2014-09-25 17:15:57,140::task::993::Storage.TaskManager.Task::(_decref) Task=`c3cde9e8-7c49-435e-8861-d598e3b16ab4`::ref 0 aborting False >Thread-69936::DEBUG::2014-09-25 17:15:57,141::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69936::DEBUG::2014-09-25 17:15:57,141::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:57,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:57,156::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69937::DEBUG::2014-09-25 17:15:57,158::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:15:57,158::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69937::DEBUG::2014-09-25 17:15:57,160::task::595::Storage.TaskManager.Task::(_updateState) Task=`44923c73-02c8-4c07-a8c0-c195f87316c8`::moving from state init -> state preparing >Thread-69937::INFO::2014-09-25 17:15:57,161::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69937::DEBUG::2014-09-25 17:15:57,161::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`67b31d09-7939-4a7d-9b35-a1c74c51ad0f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69937::DEBUG::2014-09-25 17:15:57,162::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69937::DEBUG::2014-09-25 17:15:57,162::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69937::DEBUG::2014-09-25 17:15:57,162::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`67b31d09-7939-4a7d-9b35-a1c74c51ad0f`::Granted request >Thread-69937::DEBUG::2014-09-25 17:15:57,164::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`44923c73-02c8-4c07-a8c0-c195f87316c8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69937::DEBUG::2014-09-25 17:15:57,164::task::993::Storage.TaskManager.Task::(_decref) Task=`44923c73-02c8-4c07-a8c0-c195f87316c8`::ref 1 aborting False >Thread-69937::INFO::2014-09-25 17:15:57,169::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69937::DEBUG::2014-09-25 17:15:57,169::task::1191::Storage.TaskManager.Task::(prepare) Task=`44923c73-02c8-4c07-a8c0-c195f87316c8`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69937::DEBUG::2014-09-25 17:15:57,169::task::595::Storage.TaskManager.Task::(_updateState) Task=`44923c73-02c8-4c07-a8c0-c195f87316c8`::moving from state preparing -> state finished >Thread-69937::DEBUG::2014-09-25 17:15:57,169::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69937::DEBUG::2014-09-25 17:15:57,170::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69937::DEBUG::2014-09-25 17:15:57,170::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69937::DEBUG::2014-09-25 17:15:57,170::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69937::DEBUG::2014-09-25 17:15:57,171::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69937::DEBUG::2014-09-25 17:15:57,171::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69937::DEBUG::2014-09-25 17:15:57,171::task::993::Storage.TaskManager.Task::(_decref) Task=`44923c73-02c8-4c07-a8c0-c195f87316c8`::ref 0 aborting False >Thread-69937::DEBUG::2014-09-25 17:15:57,172::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69937::DEBUG::2014-09-25 17:15:57,172::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:57,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0318464 s, 32.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:57,280::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:57,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:57,281::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:57,281::__init__::232::IOProcess::(_processLogs) (107586) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:57,306::__init__::232::IOProcess::(_processLogs) (107586) Queuing response >Thread-81::DEBUG::2014-09-25 17:15:57,307::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:15:57,338::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00273643 s, 136 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:15:57,339::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:15:57,340::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:15:57,340::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:15:57,340::__init__::232::IOProcess::(_processLogs) (107587) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:15:57,341::__init__::232::IOProcess::(_processLogs) (107587) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:15:58,009::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:15:58,010::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69938::DEBUG::2014-09-25 17:15:58,012::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:15:59,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:15:59,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0225632 s, 45.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:01,029::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:01,031::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69939::DEBUG::2014-09-25 17:16:01,033::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:01,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:01,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141851 s, 72.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:01,927::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:01,928::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:01,928::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:01,928::__init__::232::IOProcess::(_processLogs) (107588) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:01,929::__init__::232::IOProcess::(_processLogs) (107588) Queuing response >Thread-79::DEBUG::2014-09-25 17:16:01,930::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:16:01,949::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00267512 s, 210 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:01,950::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:01,950::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:01,951::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:01,951::__init__::232::IOProcess::(_processLogs) (107589) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:01,954::__init__::232::IOProcess::(_processLogs) (107589) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:01,955::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:01,956::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:01,957::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:01,957::__init__::232::IOProcess::(_processLogs) (107590) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:01,957::__init__::232::IOProcess::(_processLogs) (107590) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:01,957::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:01,958::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:01,959::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:01,959::__init__::232::IOProcess::(_processLogs) (107591) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:01,959::__init__::232::IOProcess::(_processLogs) (107591) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:01,959::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:01,960::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:01,961::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:01,961::__init__::232::IOProcess::(_processLogs) (107592) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:01,961::__init__::232::IOProcess::(_processLogs) (107592) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:02,790::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:02,791::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:02,791::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:02,791::__init__::232::IOProcess::(_processLogs) (107593) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:02,792::__init__::232::IOProcess::(_processLogs) (107593) Queuing response >Thread-80::DEBUG::2014-09-25 17:16:02,793::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:16:02,801::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000355602 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:02,803::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:02,803::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:02,803::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:02,803::__init__::232::IOProcess::(_processLogs) (107594) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:02,804::__init__::232::IOProcess::(_processLogs) (107594) Queuing response >Dummy-148::DEBUG::2014-09-25 17:16:03,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:03,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0241849 s, 42.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:04,046::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:04,048::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69940::DEBUG::2014-09-25 17:16:04,051::task::595::Storage.TaskManager.Task::(_updateState) Task=`c88471a1-61ff-4ac9-960d-85a25a3739cb`::moving from state init -> state preparing >Thread-69940::INFO::2014-09-25 17:16:04,051::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69940::INFO::2014-09-25 17:16:04,052::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00267512', 'lastCheck': '2.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000355602', 'lastCheck': '1.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00273643', 'lastCheck': '6.7', 'valid': True}} >Thread-69940::DEBUG::2014-09-25 17:16:04,052::task::1191::Storage.TaskManager.Task::(prepare) Task=`c88471a1-61ff-4ac9-960d-85a25a3739cb`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00267512', 'lastCheck': '2.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000355602', 'lastCheck': '1.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00273643', 'lastCheck': '6.7', 'valid': True}} >Thread-69940::DEBUG::2014-09-25 17:16:04,052::task::595::Storage.TaskManager.Task::(_updateState) Task=`c88471a1-61ff-4ac9-960d-85a25a3739cb`::moving from state preparing -> state finished >Thread-69940::DEBUG::2014-09-25 17:16:04,052::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69940::DEBUG::2014-09-25 17:16:04,053::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69940::DEBUG::2014-09-25 17:16:04,053::task::993::Storage.TaskManager.Task::(_decref) Task=`c88471a1-61ff-4ac9-960d-85a25a3739cb`::ref 0 aborting False >Thread-69940::DEBUG::2014-09-25 17:16:04,055::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:04,071::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:04,072::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69941::DEBUG::2014-09-25 17:16:04,074::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:05,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:05,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.21892 s, 4.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:07,120::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:07,121::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69942::DEBUG::2014-09-25 17:16:07,123::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:07,203::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69943::DEBUG::2014-09-25 17:16:07,204::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:16:07,204::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69943::DEBUG::2014-09-25 17:16:07,206::task::595::Storage.TaskManager.Task::(_updateState) Task=`5a53caf7-02ea-4dee-9df6-683bcd85fabe`::moving from state init -> state preparing >Thread-69943::INFO::2014-09-25 17:16:07,207::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69943::INFO::2014-09-25 17:16:07,207::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69943::DEBUG::2014-09-25 17:16:07,208::task::1191::Storage.TaskManager.Task::(prepare) Task=`5a53caf7-02ea-4dee-9df6-683bcd85fabe`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69943::DEBUG::2014-09-25 17:16:07,208::task::595::Storage.TaskManager.Task::(_updateState) Task=`5a53caf7-02ea-4dee-9df6-683bcd85fabe`::moving from state preparing -> state finished >Thread-69943::DEBUG::2014-09-25 17:16:07,208::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69943::DEBUG::2014-09-25 17:16:07,208::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69943::DEBUG::2014-09-25 17:16:07,208::task::993::Storage.TaskManager.Task::(_decref) Task=`5a53caf7-02ea-4dee-9df6-683bcd85fabe`::ref 0 aborting False >Thread-69943::DEBUG::2014-09-25 17:16:07,209::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69943::DEBUG::2014-09-25 17:16:07,209::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:07,222::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69944::DEBUG::2014-09-25 17:16:07,223::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:16:07,223::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69944::DEBUG::2014-09-25 17:16:07,225::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd666edb-e4ca-4e1b-aa90-1a68f26bc234`::moving from state init -> state preparing >Thread-69944::INFO::2014-09-25 17:16:07,226::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69944::DEBUG::2014-09-25 17:16:07,226::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7171b272-325c-49fc-bb4f-503fc84cfe34`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69944::DEBUG::2014-09-25 17:16:07,227::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69944::DEBUG::2014-09-25 17:16:07,227::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69944::DEBUG::2014-09-25 17:16:07,227::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7171b272-325c-49fc-bb4f-503fc84cfe34`::Granted request >Thread-69944::DEBUG::2014-09-25 17:16:07,228::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`dd666edb-e4ca-4e1b-aa90-1a68f26bc234`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69944::DEBUG::2014-09-25 17:16:07,228::task::993::Storage.TaskManager.Task::(_decref) Task=`dd666edb-e4ca-4e1b-aa90-1a68f26bc234`::ref 1 aborting False >Thread-69944::INFO::2014-09-25 17:16:07,232::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69944::DEBUG::2014-09-25 17:16:07,233::task::1191::Storage.TaskManager.Task::(prepare) Task=`dd666edb-e4ca-4e1b-aa90-1a68f26bc234`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69944::DEBUG::2014-09-25 17:16:07,233::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd666edb-e4ca-4e1b-aa90-1a68f26bc234`::moving from state preparing -> state finished >Thread-69944::DEBUG::2014-09-25 17:16:07,233::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69944::DEBUG::2014-09-25 17:16:07,233::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69944::DEBUG::2014-09-25 17:16:07,234::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69944::DEBUG::2014-09-25 17:16:07,234::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69944::DEBUG::2014-09-25 17:16:07,234::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69944::DEBUG::2014-09-25 17:16:07,235::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69944::DEBUG::2014-09-25 17:16:07,235::task::993::Storage.TaskManager.Task::(_decref) Task=`dd666edb-e4ca-4e1b-aa90-1a68f26bc234`::ref 0 aborting False >Thread-69944::DEBUG::2014-09-25 17:16:07,235::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69944::DEBUG::2014-09-25 17:16:07,236::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:16:07,342::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:07,342::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:07,343::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:07,343::__init__::232::IOProcess::(_processLogs) (107595) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:07,345::__init__::232::IOProcess::(_processLogs) (107595) Queuing response >Thread-81::DEBUG::2014-09-25 17:16:07,346::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:16:07,364::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00652847 s, 56.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:07,365::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:07,365::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:07,366::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:07,366::__init__::232::IOProcess::(_processLogs) (107596) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:07,370::__init__::232::IOProcess::(_processLogs) (107596) Queuing response >Dummy-148::DEBUG::2014-09-25 17:16:07,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:07,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194195 s, 52.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:16:09,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:09,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141849 s, 72.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:10,139::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:10,141::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69945::DEBUG::2014-09-25 17:16:10,143::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:11,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:11,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156142 s, 65.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:11,962::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:11,962::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:11,963::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:16:11,964::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:16:11,964::__init__::232::IOProcess::(_processLogs) (107597) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:11,973::__init__::232::IOProcess::(_processLogs) (107597) Queuing response >Thread-79::DEBUG::2014-09-25 17:16:11,973::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000368785 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:11,974::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:11,975::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:11,975::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:11,975::__init__::232::IOProcess::(_processLogs) (107598) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:11,976::__init__::232::IOProcess::(_processLogs) (107598) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:11,977::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:11,977::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:11,978::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:11,978::__init__::232::IOProcess::(_processLogs) (107599) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:11,978::__init__::232::IOProcess::(_processLogs) (107599) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:11,979::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:11,980::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:11,980::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:11,980::__init__::232::IOProcess::(_processLogs) (107600) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:11,980::__init__::232::IOProcess::(_processLogs) (107600) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:11,981::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:11,981::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:11,981::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:11,982::__init__::232::IOProcess::(_processLogs) (107601) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:11,982::__init__::232::IOProcess::(_processLogs) (107601) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:12,806::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:12,807::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:12,807::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:12,807::__init__::232::IOProcess::(_processLogs) (107602) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:12,807::__init__::232::IOProcess::(_processLogs) (107602) Queuing response >Thread-80::DEBUG::2014-09-25 17:16:12,809::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:16:12,817::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00051392 s, 736 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:12,818::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:12,819::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:12,819::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:12,819::__init__::232::IOProcess::(_processLogs) (107603) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:12,820::__init__::232::IOProcess::(_processLogs) (107603) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:13,158::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:13,160::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69946::DEBUG::2014-09-25 17:16:13,162::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:13,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:13,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150351 s, 68.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:16:15,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:15,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0196798 s, 52.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:16,176::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:16,180::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69947::DEBUG::2014-09-25 17:16:16,184::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:17,264::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69948::DEBUG::2014-09-25 17:16:17,264::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:16:17,264::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69948::DEBUG::2014-09-25 17:16:17,266::task::595::Storage.TaskManager.Task::(_updateState) Task=`3691f854-c5a8-4c2c-b723-965b7a21828e`::moving from state init -> state preparing >Thread-69948::INFO::2014-09-25 17:16:17,266::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69948::INFO::2014-09-25 17:16:17,266::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69948::DEBUG::2014-09-25 17:16:17,267::task::1191::Storage.TaskManager.Task::(prepare) Task=`3691f854-c5a8-4c2c-b723-965b7a21828e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69948::DEBUG::2014-09-25 17:16:17,267::task::595::Storage.TaskManager.Task::(_updateState) Task=`3691f854-c5a8-4c2c-b723-965b7a21828e`::moving from state preparing -> state finished >Thread-69948::DEBUG::2014-09-25 17:16:17,267::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69948::DEBUG::2014-09-25 17:16:17,267::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69948::DEBUG::2014-09-25 17:16:17,267::task::993::Storage.TaskManager.Task::(_decref) Task=`3691f854-c5a8-4c2c-b723-965b7a21828e`::ref 0 aborting False >Thread-69948::DEBUG::2014-09-25 17:16:17,267::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69948::DEBUG::2014-09-25 17:16:17,268::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:17,287::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:17,288::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69949::DEBUG::2014-09-25 17:16:17,288::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69949::DEBUG::2014-09-25 17:16:17,289::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f311e9e-0e9f-4a40-b866-19605bc0a0ae`::moving from state init -> state preparing >Thread-69949::INFO::2014-09-25 17:16:17,289::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69949::DEBUG::2014-09-25 17:16:17,290::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`71abe5fa-8e16-4867-9338-21aafc2c9316`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69949::DEBUG::2014-09-25 17:16:17,290::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69949::DEBUG::2014-09-25 17:16:17,290::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69949::DEBUG::2014-09-25 17:16:17,290::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`71abe5fa-8e16-4867-9338-21aafc2c9316`::Granted request >Thread-69949::DEBUG::2014-09-25 17:16:17,291::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2f311e9e-0e9f-4a40-b866-19605bc0a0ae`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69949::DEBUG::2014-09-25 17:16:17,291::task::993::Storage.TaskManager.Task::(_decref) Task=`2f311e9e-0e9f-4a40-b866-19605bc0a0ae`::ref 1 aborting False >Thread-69949::INFO::2014-09-25 17:16:17,294::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69949::DEBUG::2014-09-25 17:16:17,294::task::1191::Storage.TaskManager.Task::(prepare) Task=`2f311e9e-0e9f-4a40-b866-19605bc0a0ae`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69949::DEBUG::2014-09-25 17:16:17,294::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f311e9e-0e9f-4a40-b866-19605bc0a0ae`::moving from state preparing -> state finished >Thread-69949::DEBUG::2014-09-25 17:16:17,294::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69949::DEBUG::2014-09-25 17:16:17,294::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69949::DEBUG::2014-09-25 17:16:17,295::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69949::DEBUG::2014-09-25 17:16:17,295::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69949::DEBUG::2014-09-25 17:16:17,295::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69949::DEBUG::2014-09-25 17:16:17,295::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69949::DEBUG::2014-09-25 17:16:17,295::task::993::Storage.TaskManager.Task::(_decref) Task=`2f311e9e-0e9f-4a40-b866-19605bc0a0ae`::ref 0 aborting False >Thread-69949::DEBUG::2014-09-25 17:16:17,296::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69949::DEBUG::2014-09-25 17:16:17,296::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:16:17,371::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:17,371::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:17,372::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:17,372::__init__::232::IOProcess::(_processLogs) (107604) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:17,373::__init__::232::IOProcess::(_processLogs) (107604) Queuing response >Thread-81::DEBUG::2014-09-25 17:16:17,373::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:16:17,391::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00473916 s, 78.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:17,392::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:17,393::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:17,393::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:17,393::__init__::232::IOProcess::(_processLogs) (107605) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:17,396::__init__::232::IOProcess::(_processLogs) (107605) Queuing response >Dummy-148::DEBUG::2014-09-25 17:16:17,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:17,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0187921 s, 54.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:19,202::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:19,205::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69950::DEBUG::2014-09-25 17:16:19,207::task::595::Storage.TaskManager.Task::(_updateState) Task=`488ffd6e-87fa-46f6-bcee-68cae551d733`::moving from state init -> state preparing >Thread-69950::INFO::2014-09-25 17:16:19,208::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69950::INFO::2014-09-25 17:16:19,208::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000368785', 'lastCheck': '7.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00051392', 'lastCheck': '6.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00473916', 'lastCheck': '1.8', 'valid': True}} >Thread-69950::DEBUG::2014-09-25 17:16:19,208::task::1191::Storage.TaskManager.Task::(prepare) Task=`488ffd6e-87fa-46f6-bcee-68cae551d733`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000368785', 'lastCheck': '7.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00051392', 'lastCheck': '6.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00473916', 'lastCheck': '1.8', 'valid': True}} >Thread-69950::DEBUG::2014-09-25 17:16:19,209::task::595::Storage.TaskManager.Task::(_updateState) Task=`488ffd6e-87fa-46f6-bcee-68cae551d733`::moving from state preparing -> state finished >Thread-69950::DEBUG::2014-09-25 17:16:19,209::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69950::DEBUG::2014-09-25 17:16:19,209::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69950::DEBUG::2014-09-25 17:16:19,209::task::993::Storage.TaskManager.Task::(_decref) Task=`488ffd6e-87fa-46f6-bcee-68cae551d733`::ref 0 aborting False >Thread-69950::DEBUG::2014-09-25 17:16:19,212::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:19,233::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:19,234::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69951::DEBUG::2014-09-25 17:16:19,236::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:19,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:19,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0228137 s, 44.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:16:21,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:16:21,984::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:21,984::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:21,984::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:21,985::__init__::232::IOProcess::(_processLogs) (107606) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:21,989::__init__::232::IOProcess::(_processLogs) (107606) Queuing response >Thread-79::DEBUG::2014-09-25 17:16:21,992::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:22,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0244899 s, 41.8 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 17:16:22,018::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00911567 s, 61.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:22,020::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:22,021::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:22,022::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:22,023::__init__::232::IOProcess::(_processLogs) (107607) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:22,024::__init__::232::IOProcess::(_processLogs) (107607) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:22,027::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:22,028::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:22,029::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:22,030::__init__::232::IOProcess::(_processLogs) (107608) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:22,031::__init__::232::IOProcess::(_processLogs) (107608) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:22,032::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:22,033::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:22,034::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:22,035::__init__::232::IOProcess::(_processLogs) (107609) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:22,036::__init__::232::IOProcess::(_processLogs) (107609) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:22,038::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:22,039::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:22,041::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:22,042::__init__::232::IOProcess::(_processLogs) (107610) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:22,042::__init__::232::IOProcess::(_processLogs) (107610) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:22,305::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:22,306::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69952::DEBUG::2014-09-25 17:16:22,308::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:16:22,822::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:22,822::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:22,822::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:22,822::__init__::232::IOProcess::(_processLogs) (107611) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:22,823::__init__::232::IOProcess::(_processLogs) (107611) Queuing response >Thread-80::DEBUG::2014-09-25 17:16:22,824::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:16:22,833::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000533637 s, 708 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:22,834::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:22,834::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:22,835::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:22,835::__init__::232::IOProcess::(_processLogs) (107612) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:22,839::__init__::232::IOProcess::(_processLogs) (107612) Queuing response >Dummy-148::DEBUG::2014-09-25 17:16:24,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:24,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218251 s, 46.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:25,326::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:25,328::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69953::DEBUG::2014-09-25 17:16:25,330::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:26,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:26,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019341 s, 52.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:27,324::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69954::DEBUG::2014-09-25 17:16:27,325::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:16:27,325::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69954::DEBUG::2014-09-25 17:16:27,328::task::595::Storage.TaskManager.Task::(_updateState) Task=`e83c7c2f-96cb-4380-8c7c-0a1452045421`::moving from state init -> state preparing >Thread-69954::INFO::2014-09-25 17:16:27,328::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69954::INFO::2014-09-25 17:16:27,328::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69954::DEBUG::2014-09-25 17:16:27,329::task::1191::Storage.TaskManager.Task::(prepare) Task=`e83c7c2f-96cb-4380-8c7c-0a1452045421`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69954::DEBUG::2014-09-25 17:16:27,329::task::595::Storage.TaskManager.Task::(_updateState) Task=`e83c7c2f-96cb-4380-8c7c-0a1452045421`::moving from state preparing -> state finished >Thread-69954::DEBUG::2014-09-25 17:16:27,329::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69954::DEBUG::2014-09-25 17:16:27,329::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69954::DEBUG::2014-09-25 17:16:27,330::task::993::Storage.TaskManager.Task::(_decref) Task=`e83c7c2f-96cb-4380-8c7c-0a1452045421`::ref 0 aborting False >Thread-69954::DEBUG::2014-09-25 17:16:27,330::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69954::DEBUG::2014-09-25 17:16:27,330::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:27,341::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69955::DEBUG::2014-09-25 17:16:27,342::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:16:27,342::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69955::DEBUG::2014-09-25 17:16:27,344::task::595::Storage.TaskManager.Task::(_updateState) Task=`eb0cee93-828c-4d6d-9c28-45dd8378df2c`::moving from state init -> state preparing >Thread-69955::INFO::2014-09-25 17:16:27,344::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69955::DEBUG::2014-09-25 17:16:27,345::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d58d21fe-f8e0-41e5-acd9-b9e5111bf686`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69955::DEBUG::2014-09-25 17:16:27,345::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69955::DEBUG::2014-09-25 17:16:27,346::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69955::DEBUG::2014-09-25 17:16:27,346::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d58d21fe-f8e0-41e5-acd9-b9e5111bf686`::Granted request >Thread-69955::DEBUG::2014-09-25 17:16:27,346::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`eb0cee93-828c-4d6d-9c28-45dd8378df2c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69955::DEBUG::2014-09-25 17:16:27,347::task::993::Storage.TaskManager.Task::(_decref) Task=`eb0cee93-828c-4d6d-9c28-45dd8378df2c`::ref 1 aborting False >Thread-69955::INFO::2014-09-25 17:16:27,351::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69955::DEBUG::2014-09-25 17:16:27,351::task::1191::Storage.TaskManager.Task::(prepare) Task=`eb0cee93-828c-4d6d-9c28-45dd8378df2c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69955::DEBUG::2014-09-25 17:16:27,352::task::595::Storage.TaskManager.Task::(_updateState) Task=`eb0cee93-828c-4d6d-9c28-45dd8378df2c`::moving from state preparing -> state finished >Thread-69955::DEBUG::2014-09-25 17:16:27,352::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69955::DEBUG::2014-09-25 17:16:27,352::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69955::DEBUG::2014-09-25 17:16:27,352::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69955::DEBUG::2014-09-25 17:16:27,353::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69955::DEBUG::2014-09-25 17:16:27,353::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69955::DEBUG::2014-09-25 17:16:27,353::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69955::DEBUG::2014-09-25 17:16:27,353::task::993::Storage.TaskManager.Task::(_decref) Task=`eb0cee93-828c-4d6d-9c28-45dd8378df2c`::ref 0 aborting False >Thread-69955::DEBUG::2014-09-25 17:16:27,354::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69955::DEBUG::2014-09-25 17:16:27,354::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:16:27,397::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:27,398::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:27,398::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:27,398::__init__::232::IOProcess::(_processLogs) (107613) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:27,401::__init__::232::IOProcess::(_processLogs) (107613) Queuing response >Thread-81::DEBUG::2014-09-25 17:16:27,402::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:16:27,412::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0014805 s, 251 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:27,413::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:27,413::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:27,413::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:27,414::__init__::232::IOProcess::(_processLogs) (107614) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:27,414::__init__::232::IOProcess::(_processLogs) (107614) Queuing response >Dummy-148::DEBUG::2014-09-25 17:16:28,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:28,180::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171461 s, 59.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:28,345::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:28,346::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69956::DEBUG::2014-09-25 17:16:28,348::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:30,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:30,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201485 s, 50.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:31,363::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:31,364::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69957::DEBUG::2014-09-25 17:16:31,366::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:16:32,041::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:32,042::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:32,042::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:32,042::__init__::232::IOProcess::(_processLogs) (107615) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:32,044::__init__::232::IOProcess::(_processLogs) (107615) Queuing response >Thread-79::DEBUG::2014-09-25 17:16:32,045::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:16:32,058::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00294867 s, 191 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:32,059::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:32,059::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:32,059::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:32,060::__init__::232::IOProcess::(_processLogs) (107616) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:32,063::__init__::232::IOProcess::(_processLogs) (107616) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:32,064::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:32,065::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:32,065::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:32,066::__init__::232::IOProcess::(_processLogs) (107617) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:32,066::__init__::232::IOProcess::(_processLogs) (107617) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:32,066::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:32,067::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:32,067::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:32,067::__init__::232::IOProcess::(_processLogs) (107618) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:32,067::__init__::232::IOProcess::(_processLogs) (107618) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:32,068::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:32,069::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:32,070::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:32,070::__init__::232::IOProcess::(_processLogs) (107619) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:32,070::__init__::232::IOProcess::(_processLogs) (107619) Queuing response >Dummy-148::DEBUG::2014-09-25 17:16:32,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:32,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174757 s, 58.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:32,841::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:32,841::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:32,842::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:32,842::__init__::232::IOProcess::(_processLogs) (107620) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:32,842::__init__::232::IOProcess::(_processLogs) (107620) Queuing response >Thread-80::DEBUG::2014-09-25 17:16:32,843::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:16:32,852::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000459001 s, 824 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:32,854::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:32,854::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:32,855::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:32,856::__init__::232::IOProcess::(_processLogs) (107621) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:32,856::__init__::232::IOProcess::(_processLogs) (107621) Queuing response >Dummy-148::DEBUG::2014-09-25 17:16:34,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:34,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165786 s, 61.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:34,378::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:34,379::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69958::DEBUG::2014-09-25 17:16:34,382::task::595::Storage.TaskManager.Task::(_updateState) Task=`2b9915f6-6483-4f14-b807-a871437faae4`::moving from state init -> state preparing >Thread-69958::INFO::2014-09-25 17:16:34,383::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69958::INFO::2014-09-25 17:16:34,383::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00294867', 'lastCheck': '2.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000459001', 'lastCheck': '1.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0014805', 'lastCheck': '7.0', 'valid': True}} >Thread-69958::DEBUG::2014-09-25 17:16:34,383::task::1191::Storage.TaskManager.Task::(prepare) Task=`2b9915f6-6483-4f14-b807-a871437faae4`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00294867', 'lastCheck': '2.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000459001', 'lastCheck': '1.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0014805', 'lastCheck': '7.0', 'valid': True}} >Thread-69958::DEBUG::2014-09-25 17:16:34,384::task::595::Storage.TaskManager.Task::(_updateState) Task=`2b9915f6-6483-4f14-b807-a871437faae4`::moving from state preparing -> state finished >Thread-69958::DEBUG::2014-09-25 17:16:34,384::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69958::DEBUG::2014-09-25 17:16:34,384::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69958::DEBUG::2014-09-25 17:16:34,384::task::993::Storage.TaskManager.Task::(_decref) Task=`2b9915f6-6483-4f14-b807-a871437faae4`::ref 0 aborting False >Thread-69958::DEBUG::2014-09-25 17:16:34,387::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:34,400::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:34,401::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69959::DEBUG::2014-09-25 17:16:34,403::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:36,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:36,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0231074 s, 44.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:37,379::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:37,380::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69960::DEBUG::2014-09-25 17:16:37,380::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69960::DEBUG::2014-09-25 17:16:37,382::task::595::Storage.TaskManager.Task::(_updateState) Task=`a37b1606-25a8-4335-9296-7c28387633ed`::moving from state init -> state preparing >Thread-69960::INFO::2014-09-25 17:16:37,383::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69960::INFO::2014-09-25 17:16:37,383::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69960::DEBUG::2014-09-25 17:16:37,383::task::1191::Storage.TaskManager.Task::(prepare) Task=`a37b1606-25a8-4335-9296-7c28387633ed`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69960::DEBUG::2014-09-25 17:16:37,384::task::595::Storage.TaskManager.Task::(_updateState) Task=`a37b1606-25a8-4335-9296-7c28387633ed`::moving from state preparing -> state finished >Thread-69960::DEBUG::2014-09-25 17:16:37,384::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69960::DEBUG::2014-09-25 17:16:37,384::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69960::DEBUG::2014-09-25 17:16:37,384::task::993::Storage.TaskManager.Task::(_decref) Task=`a37b1606-25a8-4335-9296-7c28387633ed`::ref 0 aborting False >Thread-69960::DEBUG::2014-09-25 17:16:37,385::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69960::DEBUG::2014-09-25 17:16:37,385::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:37,397::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69961::DEBUG::2014-09-25 17:16:37,399::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:16:37,399::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69961::DEBUG::2014-09-25 17:16:37,401::task::595::Storage.TaskManager.Task::(_updateState) Task=`2e08e452-cd4d-4c35-91d4-51d308d6ae60`::moving from state init -> state preparing >Thread-69961::INFO::2014-09-25 17:16:37,401::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69961::DEBUG::2014-09-25 17:16:37,402::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`03a0b986-04c2-43b7-bc87-efb94e9dc3d8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69961::DEBUG::2014-09-25 17:16:37,402::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69961::DEBUG::2014-09-25 17:16:37,402::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69961::DEBUG::2014-09-25 17:16:37,403::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`03a0b986-04c2-43b7-bc87-efb94e9dc3d8`::Granted request >Thread-69961::DEBUG::2014-09-25 17:16:37,403::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2e08e452-cd4d-4c35-91d4-51d308d6ae60`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69961::DEBUG::2014-09-25 17:16:37,403::task::993::Storage.TaskManager.Task::(_decref) Task=`2e08e452-cd4d-4c35-91d4-51d308d6ae60`::ref 1 aborting False >Thread-69961::INFO::2014-09-25 17:16:37,408::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69961::DEBUG::2014-09-25 17:16:37,408::task::1191::Storage.TaskManager.Task::(prepare) Task=`2e08e452-cd4d-4c35-91d4-51d308d6ae60`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69961::DEBUG::2014-09-25 17:16:37,408::task::595::Storage.TaskManager.Task::(_updateState) Task=`2e08e452-cd4d-4c35-91d4-51d308d6ae60`::moving from state preparing -> state finished >Thread-69961::DEBUG::2014-09-25 17:16:37,409::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69961::DEBUG::2014-09-25 17:16:37,409::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69961::DEBUG::2014-09-25 17:16:37,409::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69961::DEBUG::2014-09-25 17:16:37,410::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69961::DEBUG::2014-09-25 17:16:37,410::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69961::DEBUG::2014-09-25 17:16:37,410::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69961::DEBUG::2014-09-25 17:16:37,410::task::993::Storage.TaskManager.Task::(_decref) Task=`2e08e452-cd4d-4c35-91d4-51d308d6ae60`::ref 0 aborting False >Thread-69961::DEBUG::2014-09-25 17:16:37,411::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69961::DEBUG::2014-09-25 17:16:37,411::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:16:37,416::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:37,416::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:37,417::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:37,417::__init__::232::IOProcess::(_processLogs) (107622) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:37,419::__init__::232::IOProcess::(_processLogs) (107622) Queuing response >Thread-81::DEBUG::2014-09-25 17:16:37,420::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:16:37,432::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00276929 s, 134 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:37,433::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:37,433::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:37,433::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:37,434::__init__::232::IOProcess::(_processLogs) (107623) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:37,436::__init__::232::IOProcess::(_processLogs) (107623) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:37,447::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:37,449::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69962::DEBUG::2014-09-25 17:16:37,451::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:38,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:38,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191039 s, 53.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:40,466::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:40,467::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69963::DEBUG::2014-09-25 17:16:40,469::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:40,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:40,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021726 s, 47.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:42,070::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:42,071::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:42,071::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:42,071::__init__::232::IOProcess::(_processLogs) (107624) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:42,078::__init__::232::IOProcess::(_processLogs) (107624) Queuing response >Thread-79::DEBUG::2014-09-25 17:16:42,079::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:16:42,097::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00488529 s, 115 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:42,098::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:42,098::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:42,099::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:42,099::__init__::232::IOProcess::(_processLogs) (107625) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:42,101::__init__::232::IOProcess::(_processLogs) (107625) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:42,102::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:42,102::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:42,103::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:42,103::__init__::232::IOProcess::(_processLogs) (107626) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:42,109::__init__::232::IOProcess::(_processLogs) (107626) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:42,110::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:42,110::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:42,110::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:42,111::__init__::232::IOProcess::(_processLogs) (107627) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:42,117::__init__::232::IOProcess::(_processLogs) (107627) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:42,117::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:42,118::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:42,118::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:42,118::__init__::232::IOProcess::(_processLogs) (107628) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:42,123::__init__::232::IOProcess::(_processLogs) (107628) Queuing response >Dummy-148::DEBUG::2014-09-25 17:16:42,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:42,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145044 s, 70.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:42,857::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:42,857::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:42,858::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:42,858::__init__::232::IOProcess::(_processLogs) (107629) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:42,859::__init__::232::IOProcess::(_processLogs) (107629) Queuing response >Thread-80::DEBUG::2014-09-25 17:16:42,860::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:16:42,869::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00034468 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:42,870::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:42,871::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:42,871::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:42,871::__init__::232::IOProcess::(_processLogs) (107630) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:42,872::__init__::232::IOProcess::(_processLogs) (107630) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:43,485::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:43,486::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69964::DEBUG::2014-09-25 17:16:43,488::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:44,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:44,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0261465 s, 39.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:46,504::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:46,508::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69965::DEBUG::2014-09-25 17:16:46,513::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:46,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:46,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0258229 s, 39.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:47,435::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:47,437::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69966::DEBUG::2014-09-25 17:16:47,437::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-29::DEBUG::2014-09-25 17:16:47,440::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:47,442::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:47,443::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-69966::DEBUG::2014-09-25 17:16:47,444::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ef0d3de-6dd2-4566-bd4d-54fec7762eef`::moving from state init -> state preparing >Thread-29::DEBUG::2014-09-25 17:16:47,445::__init__::232::IOProcess::(_processLogs) (107631) Got request for method 'statvfs' >Thread-69966::INFO::2014-09-25 17:16:47,445::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-29::DEBUG::2014-09-25 17:16:47,447::__init__::232::IOProcess::(_processLogs) (107631) Queuing response >Thread-69966::INFO::2014-09-25 17:16:47,449::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-81::DEBUG::2014-09-25 17:16:47,451::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-69966::DEBUG::2014-09-25 17:16:47,452::task::1191::Storage.TaskManager.Task::(prepare) Task=`4ef0d3de-6dd2-4566-bd4d-54fec7762eef`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69966::DEBUG::2014-09-25 17:16:47,461::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ef0d3de-6dd2-4566-bd4d-54fec7762eef`::moving from state preparing -> state finished >Thread-69966::DEBUG::2014-09-25 17:16:47,461::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69966::DEBUG::2014-09-25 17:16:47,461::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69966::DEBUG::2014-09-25 17:16:47,462::task::993::Storage.TaskManager.Task::(_decref) Task=`4ef0d3de-6dd2-4566-bd4d-54fec7762eef`::ref 0 aborting False >Thread-69966::DEBUG::2014-09-25 17:16:47,462::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69966::DEBUG::2014-09-25 17:16:47,462::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:47,473::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:47,474::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69967::DEBUG::2014-09-25 17:16:47,474::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-81::DEBUG::2014-09-25 17:16:47,477::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00991596 s, 37.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:47,480::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-69967::DEBUG::2014-09-25 17:16:47,480::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c92c8b5-f58f-49e8-b252-8931fd514c9e`::moving from state init -> state preparing >Thread-29::DEBUG::2014-09-25 17:16:47,481::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-69967::INFO::2014-09-25 17:16:47,482::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-29::DEBUG::2014-09-25 17:16:47,483::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:47,484::__init__::232::IOProcess::(_processLogs) (107632) Got request for method 'statvfs' >Thread-69967::DEBUG::2014-09-25 17:16:47,485::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d3c4ae01-6b1e-469e-ada4-7c0699737cb4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69967::DEBUG::2014-09-25 17:16:47,486::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-29::DEBUG::2014-09-25 17:16:47,487::__init__::232::IOProcess::(_processLogs) (107632) Queuing response >Thread-69967::DEBUG::2014-09-25 17:16:47,489::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69967::DEBUG::2014-09-25 17:16:47,490::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d3c4ae01-6b1e-469e-ada4-7c0699737cb4`::Granted request >Thread-69967::DEBUG::2014-09-25 17:16:47,490::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0c92c8b5-f58f-49e8-b252-8931fd514c9e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69967::DEBUG::2014-09-25 17:16:47,490::task::993::Storage.TaskManager.Task::(_decref) Task=`0c92c8b5-f58f-49e8-b252-8931fd514c9e`::ref 1 aborting False >Thread-69967::INFO::2014-09-25 17:16:47,495::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69967::DEBUG::2014-09-25 17:16:47,495::task::1191::Storage.TaskManager.Task::(prepare) Task=`0c92c8b5-f58f-49e8-b252-8931fd514c9e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69967::DEBUG::2014-09-25 17:16:47,495::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c92c8b5-f58f-49e8-b252-8931fd514c9e`::moving from state preparing -> state finished >Thread-69967::DEBUG::2014-09-25 17:16:47,495::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69967::DEBUG::2014-09-25 17:16:47,496::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69967::DEBUG::2014-09-25 17:16:47,496::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69967::DEBUG::2014-09-25 17:16:47,496::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69967::DEBUG::2014-09-25 17:16:47,497::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69967::DEBUG::2014-09-25 17:16:47,497::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69967::DEBUG::2014-09-25 17:16:47,497::task::993::Storage.TaskManager.Task::(_decref) Task=`0c92c8b5-f58f-49e8-b252-8931fd514c9e`::ref 0 aborting False >Thread-69967::DEBUG::2014-09-25 17:16:47,498::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69967::DEBUG::2014-09-25 17:16:47,498::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:48,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:48,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172604 s, 59.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:49,533::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:49,534::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69968::DEBUG::2014-09-25 17:16:49,537::task::595::Storage.TaskManager.Task::(_updateState) Task=`cc6f7891-6d58-4320-991d-f9c3c0ae194f`::moving from state init -> state preparing >Thread-69968::INFO::2014-09-25 17:16:49,537::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69968::INFO::2014-09-25 17:16:49,538::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00488529', 'lastCheck': '7.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00034468', 'lastCheck': '6.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00991596', 'lastCheck': '2.0', 'valid': True}} >Thread-69968::DEBUG::2014-09-25 17:16:49,538::task::1191::Storage.TaskManager.Task::(prepare) Task=`cc6f7891-6d58-4320-991d-f9c3c0ae194f`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00488529', 'lastCheck': '7.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00034468', 'lastCheck': '6.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00991596', 'lastCheck': '2.0', 'valid': True}} >Thread-69968::DEBUG::2014-09-25 17:16:49,538::task::595::Storage.TaskManager.Task::(_updateState) Task=`cc6f7891-6d58-4320-991d-f9c3c0ae194f`::moving from state preparing -> state finished >Thread-69968::DEBUG::2014-09-25 17:16:49,538::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69968::DEBUG::2014-09-25 17:16:49,539::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69968::DEBUG::2014-09-25 17:16:49,539::task::993::Storage.TaskManager.Task::(_decref) Task=`cc6f7891-6d58-4320-991d-f9c3c0ae194f`::ref 0 aborting False >Thread-69968::DEBUG::2014-09-25 17:16:49,541::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:49,559::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:49,560::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69969::DEBUG::2014-09-25 17:16:49,562::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:50,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:50,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193412 s, 52.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:52,125::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:52,125::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:52,125::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:52,126::__init__::232::IOProcess::(_processLogs) (107633) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:52,127::__init__::232::IOProcess::(_processLogs) (107633) Queuing response >Thread-79::DEBUG::2014-09-25 17:16:52,129::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:16:52,146::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00861939 s, 65.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:52,147::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:52,147::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:52,148::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:52,148::__init__::232::IOProcess::(_processLogs) (107634) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:52,153::__init__::232::IOProcess::(_processLogs) (107634) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:52,155::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:52,156::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:52,156::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:52,156::__init__::232::IOProcess::(_processLogs) (107635) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:52,157::__init__::232::IOProcess::(_processLogs) (107635) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:52,157::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:52,158::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:52,158::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:52,158::__init__::232::IOProcess::(_processLogs) (107636) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:52,159::__init__::232::IOProcess::(_processLogs) (107636) Queuing response >Thread-29::DEBUG::2014-09-25 17:16:52,159::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:52,160::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:52,161::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:52,161::__init__::232::IOProcess::(_processLogs) (107637) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:16:52,161::__init__::232::IOProcess::(_processLogs) (107637) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:52,667::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:52,668::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69970::DEBUG::2014-09-25 17:16:52,670::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:16:52,873::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:52,874::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:52,874::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:52,874::__init__::232::IOProcess::(_processLogs) (107638) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:52,875::__init__::232::IOProcess::(_processLogs) (107638) Queuing response >Thread-80::DEBUG::2014-09-25 17:16:52,876::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:16:52,887::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000628813 s, 601 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:52,888::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:52,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:52,889::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:52,889::__init__::232::IOProcess::(_processLogs) (107639) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:52,890::__init__::232::IOProcess::(_processLogs) (107639) Queuing response >Dummy-148::DEBUG::2014-09-25 17:16:52,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:53,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0265894 s, 38.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:16:55,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:55,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203253 s, 50.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:55,686::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:55,687::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69971::DEBUG::2014-09-25 17:16:55,689::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:57,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:57,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184837 s, 55.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:57,490::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:57,490::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:57,491::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:57,491::__init__::232::IOProcess::(_processLogs) (107640) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:16:57,497::__init__::232::IOProcess::(_processLogs) (107640) Queuing response >Thread-81::DEBUG::2014-09-25 17:16:57,498::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:16:57,517::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00273059 s, 136 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:16:57,518::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:16:57,519::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:16:57,519::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:16:57,519::__init__::232::IOProcess::(_processLogs) (107641) Got request for method 'statvfs' >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:57,525::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-29::DEBUG::2014-09-25 17:16:57,526::__init__::232::IOProcess::(_processLogs) (107641) Queuing response >JsonRpcServer::DEBUG::2014-09-25 17:16:57,529::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69972::DEBUG::2014-09-25 17:16:57,529::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69972::DEBUG::2014-09-25 17:16:57,531::task::595::Storage.TaskManager.Task::(_updateState) Task=`26c27c0c-b3cc-4257-8b74-32301d9e3aa9`::moving from state init -> state preparing >Thread-69972::INFO::2014-09-25 17:16:57,532::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69972::INFO::2014-09-25 17:16:57,532::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69972::DEBUG::2014-09-25 17:16:57,532::task::1191::Storage.TaskManager.Task::(prepare) Task=`26c27c0c-b3cc-4257-8b74-32301d9e3aa9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69972::DEBUG::2014-09-25 17:16:57,533::task::595::Storage.TaskManager.Task::(_updateState) Task=`26c27c0c-b3cc-4257-8b74-32301d9e3aa9`::moving from state preparing -> state finished >Thread-69972::DEBUG::2014-09-25 17:16:57,533::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69972::DEBUG::2014-09-25 17:16:57,533::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69972::DEBUG::2014-09-25 17:16:57,533::task::993::Storage.TaskManager.Task::(_decref) Task=`26c27c0c-b3cc-4257-8b74-32301d9e3aa9`::ref 0 aborting False >Thread-69972::DEBUG::2014-09-25 17:16:57,534::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69972::DEBUG::2014-09-25 17:16:57,534::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:57,544::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69973::DEBUG::2014-09-25 17:16:57,545::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:16:57,545::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69973::DEBUG::2014-09-25 17:16:57,547::task::595::Storage.TaskManager.Task::(_updateState) Task=`7cd444aa-71c6-44db-af5b-f9dfead1f9ad`::moving from state init -> state preparing >Thread-69973::INFO::2014-09-25 17:16:57,547::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69973::DEBUG::2014-09-25 17:16:57,548::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c3a0118c-f698-409e-9c00-cdf1a5856af2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69973::DEBUG::2014-09-25 17:16:57,548::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69973::DEBUG::2014-09-25 17:16:57,549::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69973::DEBUG::2014-09-25 17:16:57,549::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c3a0118c-f698-409e-9c00-cdf1a5856af2`::Granted request >Thread-69973::DEBUG::2014-09-25 17:16:57,550::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7cd444aa-71c6-44db-af5b-f9dfead1f9ad`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69973::DEBUG::2014-09-25 17:16:57,550::task::993::Storage.TaskManager.Task::(_decref) Task=`7cd444aa-71c6-44db-af5b-f9dfead1f9ad`::ref 1 aborting False >Thread-69973::INFO::2014-09-25 17:16:57,554::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69973::DEBUG::2014-09-25 17:16:57,554::task::1191::Storage.TaskManager.Task::(prepare) Task=`7cd444aa-71c6-44db-af5b-f9dfead1f9ad`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69973::DEBUG::2014-09-25 17:16:57,555::task::595::Storage.TaskManager.Task::(_updateState) Task=`7cd444aa-71c6-44db-af5b-f9dfead1f9ad`::moving from state preparing -> state finished >Thread-69973::DEBUG::2014-09-25 17:16:57,555::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69973::DEBUG::2014-09-25 17:16:57,555::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69973::DEBUG::2014-09-25 17:16:57,555::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69973::DEBUG::2014-09-25 17:16:57,556::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69973::DEBUG::2014-09-25 17:16:57,556::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69973::DEBUG::2014-09-25 17:16:57,556::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69973::DEBUG::2014-09-25 17:16:57,557::task::993::Storage.TaskManager.Task::(_decref) Task=`7cd444aa-71c6-44db-af5b-f9dfead1f9ad`::ref 0 aborting False >Thread-69973::DEBUG::2014-09-25 17:16:57,557::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69973::DEBUG::2014-09-25 17:16:57,557::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:16:58,705::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:16:58,707::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69974::DEBUG::2014-09-25 17:16:58,708::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:16:59,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:16:59,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0228865 s, 44.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:17:01,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:01,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0216153 s, 47.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:01,723::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:01,724::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69975::DEBUG::2014-09-25 17:17:01,728::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:17:02,161::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:02,161::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:02,162::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:02,162::__init__::232::IOProcess::(_processLogs) (107642) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:02,163::__init__::232::IOProcess::(_processLogs) (107642) Queuing response >Thread-79::DEBUG::2014-09-25 17:17:02,164::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:17:02,181::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00128517 s, 437 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:02,182::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:02,182::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:02,182::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:02,183::__init__::232::IOProcess::(_processLogs) (107643) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:02,187::__init__::232::IOProcess::(_processLogs) (107643) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:02,189::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:02,189::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:02,190::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:02,190::__init__::232::IOProcess::(_processLogs) (107644) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:02,191::__init__::232::IOProcess::(_processLogs) (107644) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:02,191::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:02,191::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:02,192::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:02,193::__init__::232::IOProcess::(_processLogs) (107645) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:02,193::__init__::232::IOProcess::(_processLogs) (107645) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:02,193::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:02,194::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:02,194::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:02,195::__init__::232::IOProcess::(_processLogs) (107646) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:02,195::__init__::232::IOProcess::(_processLogs) (107646) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:02,892::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:02,892::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:02,893::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:02,893::__init__::232::IOProcess::(_processLogs) (107647) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:02,893::__init__::232::IOProcess::(_processLogs) (107647) Queuing response >Thread-80::DEBUG::2014-09-25 17:17:02,895::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:17:02,904::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000404462 s, 935 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:02,905::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:02,906::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:02,907::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:02,907::__init__::232::IOProcess::(_processLogs) (107648) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:02,908::__init__::232::IOProcess::(_processLogs) (107648) Queuing response >Dummy-148::DEBUG::2014-09-25 17:17:03,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:03,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134644 s, 76.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:04,738::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:04,739::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69976::DEBUG::2014-09-25 17:17:04,742::task::595::Storage.TaskManager.Task::(_updateState) Task=`0d5f3b80-e1d0-4c78-b2d2-4619feb8cddd`::moving from state init -> state preparing >Thread-69976::INFO::2014-09-25 17:17:04,743::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69976::INFO::2014-09-25 17:17:04,743::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00128517', 'lastCheck': '2.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000404462', 'lastCheck': '1.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00273059', 'lastCheck': '7.2', 'valid': True}} >Thread-69976::DEBUG::2014-09-25 17:17:04,744::task::1191::Storage.TaskManager.Task::(prepare) Task=`0d5f3b80-e1d0-4c78-b2d2-4619feb8cddd`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00128517', 'lastCheck': '2.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000404462', 'lastCheck': '1.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00273059', 'lastCheck': '7.2', 'valid': True}} >Thread-69976::DEBUG::2014-09-25 17:17:04,744::task::595::Storage.TaskManager.Task::(_updateState) Task=`0d5f3b80-e1d0-4c78-b2d2-4619feb8cddd`::moving from state preparing -> state finished >Thread-69976::DEBUG::2014-09-25 17:17:04,744::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69976::DEBUG::2014-09-25 17:17:04,744::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69976::DEBUG::2014-09-25 17:17:04,744::task::993::Storage.TaskManager.Task::(_decref) Task=`0d5f3b80-e1d0-4c78-b2d2-4619feb8cddd`::ref 0 aborting False >Thread-69976::DEBUG::2014-09-25 17:17:04,747::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:04,763::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:04,764::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69977::DEBUG::2014-09-25 17:17:04,766::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:05,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:05,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164839 s, 62.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:17:07,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:07,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158611 s, 64.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:07,529::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:07,529::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:07,530::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:07,530::__init__::232::IOProcess::(_processLogs) (107649) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:07,532::__init__::232::IOProcess::(_processLogs) (107649) Queuing response >Thread-81::DEBUG::2014-09-25 17:17:07,533::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:17:07,548::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00251552 s, 147 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:07,550::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:07,550::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:07,551::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:07,551::__init__::232::IOProcess::(_processLogs) (107650) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:07,555::__init__::232::IOProcess::(_processLogs) (107650) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:07,581::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:07,582::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69978::DEBUG::2014-09-25 17:17:07,582::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69978::DEBUG::2014-09-25 17:17:07,584::task::595::Storage.TaskManager.Task::(_updateState) Task=`b1f2700c-6960-47e3-8f90-cf07823003fa`::moving from state init -> state preparing >Thread-69978::INFO::2014-09-25 17:17:07,584::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69978::INFO::2014-09-25 17:17:07,585::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69978::DEBUG::2014-09-25 17:17:07,585::task::1191::Storage.TaskManager.Task::(prepare) Task=`b1f2700c-6960-47e3-8f90-cf07823003fa`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69978::DEBUG::2014-09-25 17:17:07,585::task::595::Storage.TaskManager.Task::(_updateState) Task=`b1f2700c-6960-47e3-8f90-cf07823003fa`::moving from state preparing -> state finished >Thread-69978::DEBUG::2014-09-25 17:17:07,586::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69978::DEBUG::2014-09-25 17:17:07,586::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69978::DEBUG::2014-09-25 17:17:07,586::task::993::Storage.TaskManager.Task::(_decref) Task=`b1f2700c-6960-47e3-8f90-cf07823003fa`::ref 0 aborting False >Thread-69978::DEBUG::2014-09-25 17:17:07,587::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69978::DEBUG::2014-09-25 17:17:07,587::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:07,599::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:07,601::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69979::DEBUG::2014-09-25 17:17:07,601::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69979::DEBUG::2014-09-25 17:17:07,603::task::595::Storage.TaskManager.Task::(_updateState) Task=`690e6259-50db-4ef4-a7cc-c61c961d2a02`::moving from state init -> state preparing >Thread-69979::INFO::2014-09-25 17:17:07,603::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69979::DEBUG::2014-09-25 17:17:07,604::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e796e3b2-4903-45e4-b211-dcd62fe3b95d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69979::DEBUG::2014-09-25 17:17:07,604::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69979::DEBUG::2014-09-25 17:17:07,604::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69979::DEBUG::2014-09-25 17:17:07,604::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e796e3b2-4903-45e4-b211-dcd62fe3b95d`::Granted request >Thread-69979::DEBUG::2014-09-25 17:17:07,605::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`690e6259-50db-4ef4-a7cc-c61c961d2a02`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69979::DEBUG::2014-09-25 17:17:07,605::task::993::Storage.TaskManager.Task::(_decref) Task=`690e6259-50db-4ef4-a7cc-c61c961d2a02`::ref 1 aborting False >Thread-69979::INFO::2014-09-25 17:17:07,610::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69979::DEBUG::2014-09-25 17:17:07,610::task::1191::Storage.TaskManager.Task::(prepare) Task=`690e6259-50db-4ef4-a7cc-c61c961d2a02`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69979::DEBUG::2014-09-25 17:17:07,610::task::595::Storage.TaskManager.Task::(_updateState) Task=`690e6259-50db-4ef4-a7cc-c61c961d2a02`::moving from state preparing -> state finished >Thread-69979::DEBUG::2014-09-25 17:17:07,610::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69979::DEBUG::2014-09-25 17:17:07,611::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69979::DEBUG::2014-09-25 17:17:07,611::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69979::DEBUG::2014-09-25 17:17:07,611::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69979::DEBUG::2014-09-25 17:17:07,611::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69979::DEBUG::2014-09-25 17:17:07,612::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69979::DEBUG::2014-09-25 17:17:07,612::task::993::Storage.TaskManager.Task::(_decref) Task=`690e6259-50db-4ef4-a7cc-c61c961d2a02`::ref 0 aborting False >Thread-69979::DEBUG::2014-09-25 17:17:07,612::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69979::DEBUG::2014-09-25 17:17:07,613::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:07,812::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:07,813::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69980::DEBUG::2014-09-25 17:17:07,815::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:09,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:09,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0190716 s, 53.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:10,832::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:10,834::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69981::DEBUG::2014-09-25 17:17:10,836::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:11,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:11,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134489 s, 76.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:12,196::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:12,196::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:12,197::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:12,197::__init__::232::IOProcess::(_processLogs) (107651) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:12,203::__init__::232::IOProcess::(_processLogs) (107651) Queuing response >Thread-79::DEBUG::2014-09-25 17:17:12,205::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:17:12,221::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00320494 s, 175 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:12,222::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:12,222::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:12,222::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:12,223::__init__::232::IOProcess::(_processLogs) (107652) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:12,226::__init__::232::IOProcess::(_processLogs) (107652) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:12,227::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:12,228::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:12,228::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:12,228::__init__::232::IOProcess::(_processLogs) (107653) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:12,228::__init__::232::IOProcess::(_processLogs) (107653) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:12,229::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:12,230::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:12,230::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:12,230::__init__::232::IOProcess::(_processLogs) (107654) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:12,230::__init__::232::IOProcess::(_processLogs) (107654) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:12,231::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:12,232::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:12,233::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:12,233::__init__::232::IOProcess::(_processLogs) (107655) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:12,234::__init__::232::IOProcess::(_processLogs) (107655) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:12,908::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:12,908::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:12,908::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:12,909::__init__::232::IOProcess::(_processLogs) (107656) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:12,909::__init__::232::IOProcess::(_processLogs) (107656) Queuing response >Thread-80::DEBUG::2014-09-25 17:17:12,910::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:17:12,920::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000414064 s, 913 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:12,921::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:12,922::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:12,922::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:12,922::__init__::232::IOProcess::(_processLogs) (107657) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:12,923::__init__::232::IOProcess::(_processLogs) (107657) Queuing response >Dummy-148::DEBUG::2014-09-25 17:17:13,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:13,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174982 s, 58.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:13,854::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:13,856::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69982::DEBUG::2014-09-25 17:17:13,858::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:15,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:15,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167588 s, 61.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:16,873::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:16,877::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69983::DEBUG::2014-09-25 17:17:16,882::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:17:17,557::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:17,557::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:17,558::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:17,558::__init__::232::IOProcess::(_processLogs) (107658) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:17,560::__init__::232::IOProcess::(_processLogs) (107658) Queuing response >Thread-81::DEBUG::2014-09-25 17:17:17,561::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:17:17,581::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00509688 s, 72.8 kB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:17:17,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:17:17,590::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:17,591::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:17,591::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:17,591::__init__::232::IOProcess::(_processLogs) (107659) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:17,592::__init__::232::IOProcess::(_processLogs) (107659) Queuing response >Dummy-148::DEBUG::2014-09-25 17:17:17,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165468 s, 61.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:17,637::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:17,639::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69984::DEBUG::2014-09-25 17:17:17,639::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69984::DEBUG::2014-09-25 17:17:17,641::task::595::Storage.TaskManager.Task::(_updateState) Task=`1ab23919-734e-412f-b6b8-2b51db09a191`::moving from state init -> state preparing >Thread-69984::INFO::2014-09-25 17:17:17,641::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69984::INFO::2014-09-25 17:17:17,642::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69984::DEBUG::2014-09-25 17:17:17,642::task::1191::Storage.TaskManager.Task::(prepare) Task=`1ab23919-734e-412f-b6b8-2b51db09a191`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69984::DEBUG::2014-09-25 17:17:17,642::task::595::Storage.TaskManager.Task::(_updateState) Task=`1ab23919-734e-412f-b6b8-2b51db09a191`::moving from state preparing -> state finished >Thread-69984::DEBUG::2014-09-25 17:17:17,643::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69984::DEBUG::2014-09-25 17:17:17,643::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69984::DEBUG::2014-09-25 17:17:17,643::task::993::Storage.TaskManager.Task::(_decref) Task=`1ab23919-734e-412f-b6b8-2b51db09a191`::ref 0 aborting False >Thread-69984::DEBUG::2014-09-25 17:17:17,643::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69984::DEBUG::2014-09-25 17:17:17,644::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:17,655::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69985::DEBUG::2014-09-25 17:17:17,656::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:17:17,656::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69985::DEBUG::2014-09-25 17:17:17,658::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c422dcc-fa07-4342-91fc-62b4c317f07f`::moving from state init -> state preparing >Thread-69985::INFO::2014-09-25 17:17:17,658::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69985::DEBUG::2014-09-25 17:17:17,659::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cec01c7b-e408-4260-9657-45eb98009039`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69985::DEBUG::2014-09-25 17:17:17,659::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69985::DEBUG::2014-09-25 17:17:17,660::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69985::DEBUG::2014-09-25 17:17:17,660::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cec01c7b-e408-4260-9657-45eb98009039`::Granted request >Thread-69985::DEBUG::2014-09-25 17:17:17,661::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3c422dcc-fa07-4342-91fc-62b4c317f07f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69985::DEBUG::2014-09-25 17:17:17,661::task::993::Storage.TaskManager.Task::(_decref) Task=`3c422dcc-fa07-4342-91fc-62b4c317f07f`::ref 1 aborting False >Thread-69985::INFO::2014-09-25 17:17:17,665::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69985::DEBUG::2014-09-25 17:17:17,665::task::1191::Storage.TaskManager.Task::(prepare) Task=`3c422dcc-fa07-4342-91fc-62b4c317f07f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69985::DEBUG::2014-09-25 17:17:17,666::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c422dcc-fa07-4342-91fc-62b4c317f07f`::moving from state preparing -> state finished >Thread-69985::DEBUG::2014-09-25 17:17:17,666::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69985::DEBUG::2014-09-25 17:17:17,666::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69985::DEBUG::2014-09-25 17:17:17,666::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69985::DEBUG::2014-09-25 17:17:17,667::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69985::DEBUG::2014-09-25 17:17:17,667::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69985::DEBUG::2014-09-25 17:17:17,667::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69985::DEBUG::2014-09-25 17:17:17,668::task::993::Storage.TaskManager.Task::(_decref) Task=`3c422dcc-fa07-4342-91fc-62b4c317f07f`::ref 0 aborting False >Thread-69985::DEBUG::2014-09-25 17:17:17,668::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69985::DEBUG::2014-09-25 17:17:17,668::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:19,629::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:19,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131597 s, 77.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:19,897::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:19,899::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69986::DEBUG::2014-09-25 17:17:19,901::task::595::Storage.TaskManager.Task::(_updateState) Task=`e9e17c5f-1260-4fbb-a657-bb60f2f636cf`::moving from state init -> state preparing >Thread-69986::INFO::2014-09-25 17:17:19,902::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69986::INFO::2014-09-25 17:17:19,902::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00320494', 'lastCheck': '7.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000414064', 'lastCheck': '7.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00509688', 'lastCheck': '2.3', 'valid': True}} >Thread-69986::DEBUG::2014-09-25 17:17:19,903::task::1191::Storage.TaskManager.Task::(prepare) Task=`e9e17c5f-1260-4fbb-a657-bb60f2f636cf`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00320494', 'lastCheck': '7.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000414064', 'lastCheck': '7.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00509688', 'lastCheck': '2.3', 'valid': True}} >Thread-69986::DEBUG::2014-09-25 17:17:19,903::task::595::Storage.TaskManager.Task::(_updateState) Task=`e9e17c5f-1260-4fbb-a657-bb60f2f636cf`::moving from state preparing -> state finished >Thread-69986::DEBUG::2014-09-25 17:17:19,903::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69986::DEBUG::2014-09-25 17:17:19,903::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69986::DEBUG::2014-09-25 17:17:19,903::task::993::Storage.TaskManager.Task::(_decref) Task=`e9e17c5f-1260-4fbb-a657-bb60f2f636cf`::ref 0 aborting False >Thread-69986::DEBUG::2014-09-25 17:17:19,906::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:19,926::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:19,928::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69987::DEBUG::2014-09-25 17:17:19,929::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:21,671::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:21,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164293 s, 62.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:22,234::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:22,234::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:22,235::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:22,235::__init__::232::IOProcess::(_processLogs) (107660) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:22,236::__init__::232::IOProcess::(_processLogs) (107660) Queuing response >Thread-79::DEBUG::2014-09-25 17:17:22,238::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:17:22,253::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00353426 s, 159 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:22,254::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:22,254::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:22,255::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:22,255::__init__::232::IOProcess::(_processLogs) (107661) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:22,257::__init__::232::IOProcess::(_processLogs) (107661) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:22,259::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:22,259::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:22,259::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:22,260::__init__::232::IOProcess::(_processLogs) (107662) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:22,260::__init__::232::IOProcess::(_processLogs) (107662) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:22,261::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:22,261::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:22,262::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:22,262::__init__::232::IOProcess::(_processLogs) (107663) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:22,262::__init__::232::IOProcess::(_processLogs) (107663) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:22,263::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:22,263::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:22,264::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:22,265::__init__::232::IOProcess::(_processLogs) (107664) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:22,265::__init__::232::IOProcess::(_processLogs) (107664) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:22,925::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:22,925::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:22,926::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:22,926::__init__::232::IOProcess::(_processLogs) (107665) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:22,926::__init__::232::IOProcess::(_processLogs) (107665) Queuing response >Thread-80::DEBUG::2014-09-25 17:17:22,928::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:17:22,936::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000375838 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:22,938::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:22,938::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:22,938::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:22,940::__init__::232::IOProcess::(_processLogs) (107666) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:22,940::__init__::232::IOProcess::(_processLogs) (107666) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:22,990::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:22,992::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69988::DEBUG::2014-09-25 17:17:22,994::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:23,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:23,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137128 s, 74.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:17:25,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:25,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173288 s, 59.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:26,007::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:26,009::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69989::DEBUG::2014-09-25 17:17:26,011::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 17:17:27,594::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:17:27,598::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:27,598::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:27,598::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:27,599::__init__::232::IOProcess::(_processLogs) (107667) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:17:27,599::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:27,599::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:27,599::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:27,600::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:27,600::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:27,600::__init__::232::IOProcess::(_processLogs) (107668) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:17:27,600::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:27,601::__init__::232::IOProcess::(_processLogs) (107669) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:17:27,601::__init__::232::IOProcess::(_processLogs) (107667) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:27,602::__init__::232::IOProcess::(_processLogs) (107668) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:27,604::__init__::232::IOProcess::(_processLogs) (107669) Queuing response >Thread-81::DEBUG::2014-09-25 17:17:27,609::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:17:27,610::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:27,610::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:27,610::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:27,611::__init__::232::IOProcess::(_processLogs) (107670) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:17:27,645::__init__::232::IOProcess::(_processLogs) (107670) Queuing response >Thread-81::DEBUG::2014-09-25 17:17:27,646::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:17:27,647::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:27,648::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:27,648::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:27,648::__init__::232::IOProcess::(_processLogs) (107671) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:17:27,655::__init__::232::IOProcess::(_processLogs) (107671) Queuing response >Thread-81::DEBUG::2014-09-25 17:17:27,655::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:17:27,656::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:27,656::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:27,657::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:27,657::__init__::232::IOProcess::(_processLogs) (107672) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:27,658::__init__::232::IOProcess::(_processLogs) (107672) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:27,658::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:27,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:27,659::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:27,659::__init__::232::IOProcess::(_processLogs) (107673) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:17:27,667::__init__::232::IOProcess::(_processLogs) (107673) Queuing response >Thread-81::DEBUG::2014-09-25 17:17:27,668::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 17:17:27,668::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:17:27,668::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:17:27,669::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:27,669::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:27,669::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:27,670::__init__::232::IOProcess::(_processLogs) (107674) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:27,671::__init__::232::IOProcess::(_processLogs) (107674) Queuing response >Thread-81::DEBUG::2014-09-25 17:17:27,671::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:17:27,681::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000964327 s, 385 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:27,682::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:27,683::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:27,683::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:27,683::__init__::232::IOProcess::(_processLogs) (107675) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:27,685::__init__::232::IOProcess::(_processLogs) (107675) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:27,698::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:27,700::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69993::DEBUG::2014-09-25 17:17:27,700::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69993::DEBUG::2014-09-25 17:17:27,702::task::595::Storage.TaskManager.Task::(_updateState) Task=`f942794e-71d2-4cd5-a641-01f6ad09e286`::moving from state init -> state preparing >Thread-69993::INFO::2014-09-25 17:17:27,702::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69993::INFO::2014-09-25 17:17:27,703::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69993::DEBUG::2014-09-25 17:17:27,703::task::1191::Storage.TaskManager.Task::(prepare) Task=`f942794e-71d2-4cd5-a641-01f6ad09e286`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69993::DEBUG::2014-09-25 17:17:27,703::task::595::Storage.TaskManager.Task::(_updateState) Task=`f942794e-71d2-4cd5-a641-01f6ad09e286`::moving from state preparing -> state finished >Thread-69993::DEBUG::2014-09-25 17:17:27,704::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69993::DEBUG::2014-09-25 17:17:27,704::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69993::DEBUG::2014-09-25 17:17:27,704::task::993::Storage.TaskManager.Task::(_decref) Task=`f942794e-71d2-4cd5-a641-01f6ad09e286`::ref 0 aborting False >Thread-69993::DEBUG::2014-09-25 17:17:27,704::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69993::DEBUG::2014-09-25 17:17:27,705::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:27,718::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:27,719::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69994::DEBUG::2014-09-25 17:17:27,719::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-69994::DEBUG::2014-09-25 17:17:27,721::task::595::Storage.TaskManager.Task::(_updateState) Task=`39d91a85-3d53-4148-a2d6-a4665a947264`::moving from state init -> state preparing >Thread-69994::INFO::2014-09-25 17:17:27,721::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69994::DEBUG::2014-09-25 17:17:27,722::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4ae33812-4e88-48be-8001-1d741e2989c0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-69994::DEBUG::2014-09-25 17:17:27,722::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-69994::DEBUG::2014-09-25 17:17:27,723::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-69994::DEBUG::2014-09-25 17:17:27,723::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4ae33812-4e88-48be-8001-1d741e2989c0`::Granted request >Thread-69994::DEBUG::2014-09-25 17:17:27,723::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`39d91a85-3d53-4148-a2d6-a4665a947264`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-69994::DEBUG::2014-09-25 17:17:27,724::task::993::Storage.TaskManager.Task::(_decref) Task=`39d91a85-3d53-4148-a2d6-a4665a947264`::ref 1 aborting False >Thread-69994::INFO::2014-09-25 17:17:27,728::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69994::DEBUG::2014-09-25 17:17:27,728::task::1191::Storage.TaskManager.Task::(prepare) Task=`39d91a85-3d53-4148-a2d6-a4665a947264`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69994::DEBUG::2014-09-25 17:17:27,728::task::595::Storage.TaskManager.Task::(_updateState) Task=`39d91a85-3d53-4148-a2d6-a4665a947264`::moving from state preparing -> state finished >Thread-69994::DEBUG::2014-09-25 17:17:27,729::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-69994::DEBUG::2014-09-25 17:17:27,729::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69994::DEBUG::2014-09-25 17:17:27,729::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-69994::DEBUG::2014-09-25 17:17:27,729::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-69994::DEBUG::2014-09-25 17:17:27,730::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-69994::DEBUG::2014-09-25 17:17:27,730::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-69994::DEBUG::2014-09-25 17:17:27,730::task::993::Storage.TaskManager.Task::(_decref) Task=`39d91a85-3d53-4148-a2d6-a4665a947264`::ref 0 aborting False >Thread-69994::DEBUG::2014-09-25 17:17:27,731::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-69994::DEBUG::2014-09-25 17:17:27,731::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:27,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:27,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146251 s, 70.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:29,029::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:29,031::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69995::DEBUG::2014-09-25 17:17:29,033::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:29,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:29,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194261 s, 52.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:17:31,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:31,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178809 s, 57.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:32,050::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:32,051::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69996::DEBUG::2014-09-25 17:17:32,053::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:17:32,265::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:32,266::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:32,266::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:32,266::__init__::232::IOProcess::(_processLogs) (107676) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:32,267::__init__::232::IOProcess::(_processLogs) (107676) Queuing response >Thread-79::DEBUG::2014-09-25 17:17:32,268::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:17:32,276::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000342 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:32,277::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:32,278::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:32,278::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:32,278::__init__::232::IOProcess::(_processLogs) (107677) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:32,279::__init__::232::IOProcess::(_processLogs) (107677) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:32,280::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:32,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:32,281::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:32,282::__init__::232::IOProcess::(_processLogs) (107678) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:32,282::__init__::232::IOProcess::(_processLogs) (107678) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:32,282::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:32,283::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:32,283::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:32,284::__init__::232::IOProcess::(_processLogs) (107679) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:32,284::__init__::232::IOProcess::(_processLogs) (107679) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:32,284::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:32,285::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:32,285::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:32,286::__init__::232::IOProcess::(_processLogs) (107680) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:32,286::__init__::232::IOProcess::(_processLogs) (107680) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:32,941::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:32,941::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:32,941::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:32,942::__init__::232::IOProcess::(_processLogs) (107681) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:32,942::__init__::232::IOProcess::(_processLogs) (107681) Queuing response >Thread-80::DEBUG::2014-09-25 17:17:32,943::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:17:32,952::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000626573 s, 603 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:32,953::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:32,953::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:32,953::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:32,954::__init__::232::IOProcess::(_processLogs) (107682) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:32,954::__init__::232::IOProcess::(_processLogs) (107682) Queuing response >Dummy-148::DEBUG::2014-09-25 17:17:33,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:33,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150048 s, 68.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:35,070::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:35,071::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69997::DEBUG::2014-09-25 17:17:35,075::task::595::Storage.TaskManager.Task::(_updateState) Task=`e1f90b1a-eaf5-4519-986d-d235079409cc`::moving from state init -> state preparing >Thread-69997::INFO::2014-09-25 17:17:35,075::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-69997::INFO::2014-09-25 17:17:35,075::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000342', 'lastCheck': '2.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000626573', 'lastCheck': '2.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000964327', 'lastCheck': '7.4', 'valid': True}} >Thread-69997::DEBUG::2014-09-25 17:17:35,076::task::1191::Storage.TaskManager.Task::(prepare) Task=`e1f90b1a-eaf5-4519-986d-d235079409cc`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000342', 'lastCheck': '2.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000626573', 'lastCheck': '2.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000964327', 'lastCheck': '7.4', 'valid': True}} >Thread-69997::DEBUG::2014-09-25 17:17:35,076::task::595::Storage.TaskManager.Task::(_updateState) Task=`e1f90b1a-eaf5-4519-986d-d235079409cc`::moving from state preparing -> state finished >Thread-69997::DEBUG::2014-09-25 17:17:35,076::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69997::DEBUG::2014-09-25 17:17:35,076::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69997::DEBUG::2014-09-25 17:17:35,077::task::993::Storage.TaskManager.Task::(_decref) Task=`e1f90b1a-eaf5-4519-986d-d235079409cc`::ref 0 aborting False >Thread-69997::DEBUG::2014-09-25 17:17:35,079::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:35,100::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:35,101::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69998::DEBUG::2014-09-25 17:17:35,103::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:35,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:36,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214975 s, 47.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:37,687::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:37,687::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:37,687::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:37,688::__init__::232::IOProcess::(_processLogs) (107683) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:37,688::__init__::232::IOProcess::(_processLogs) (107683) Queuing response >Thread-81::DEBUG::2014-09-25 17:17:37,690::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:17:37,698::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000449153 s, 826 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:37,699::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:37,700::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:37,701::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:37,702::__init__::232::IOProcess::(_processLogs) (107684) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:37,702::__init__::232::IOProcess::(_processLogs) (107684) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:37,758::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-69999::DEBUG::2014-09-25 17:17:37,760::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:17:37,760::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-69999::DEBUG::2014-09-25 17:17:37,762::task::595::Storage.TaskManager.Task::(_updateState) Task=`a9a8f9a5-7d26-4308-9350-9319ba66f064`::moving from state init -> state preparing >Thread-69999::INFO::2014-09-25 17:17:37,762::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-69999::INFO::2014-09-25 17:17:37,763::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69999::DEBUG::2014-09-25 17:17:37,763::task::1191::Storage.TaskManager.Task::(prepare) Task=`a9a8f9a5-7d26-4308-9350-9319ba66f064`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-69999::DEBUG::2014-09-25 17:17:37,763::task::595::Storage.TaskManager.Task::(_updateState) Task=`a9a8f9a5-7d26-4308-9350-9319ba66f064`::moving from state preparing -> state finished >Thread-69999::DEBUG::2014-09-25 17:17:37,764::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-69999::DEBUG::2014-09-25 17:17:37,764::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-69999::DEBUG::2014-09-25 17:17:37,764::task::993::Storage.TaskManager.Task::(_decref) Task=`a9a8f9a5-7d26-4308-9350-9319ba66f064`::ref 0 aborting False >Thread-69999::DEBUG::2014-09-25 17:17:37,764::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-69999::DEBUG::2014-09-25 17:17:37,765::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:37,777::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70000::DEBUG::2014-09-25 17:17:37,779::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:17:37,779::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70000::DEBUG::2014-09-25 17:17:37,781::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc5077c8-2a97-49bd-b359-0fd37ef4794f`::moving from state init -> state preparing >Thread-70000::INFO::2014-09-25 17:17:37,781::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70000::DEBUG::2014-09-25 17:17:37,782::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ffbd4a02-69b5-4d75-9ad1-bb9b9e9d6f44`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70000::DEBUG::2014-09-25 17:17:37,782::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70000::DEBUG::2014-09-25 17:17:37,782::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70000::DEBUG::2014-09-25 17:17:37,783::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ffbd4a02-69b5-4d75-9ad1-bb9b9e9d6f44`::Granted request >Thread-70000::DEBUG::2014-09-25 17:17:37,783::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`dc5077c8-2a97-49bd-b359-0fd37ef4794f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70000::DEBUG::2014-09-25 17:17:37,783::task::993::Storage.TaskManager.Task::(_decref) Task=`dc5077c8-2a97-49bd-b359-0fd37ef4794f`::ref 1 aborting False >Thread-70000::INFO::2014-09-25 17:17:37,788::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70000::DEBUG::2014-09-25 17:17:37,788::task::1191::Storage.TaskManager.Task::(prepare) Task=`dc5077c8-2a97-49bd-b359-0fd37ef4794f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70000::DEBUG::2014-09-25 17:17:37,788::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc5077c8-2a97-49bd-b359-0fd37ef4794f`::moving from state preparing -> state finished >Thread-70000::DEBUG::2014-09-25 17:17:37,789::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70000::DEBUG::2014-09-25 17:17:37,789::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70000::DEBUG::2014-09-25 17:17:37,789::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70000::DEBUG::2014-09-25 17:17:37,790::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70000::DEBUG::2014-09-25 17:17:37,790::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70000::DEBUG::2014-09-25 17:17:37,790::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70000::DEBUG::2014-09-25 17:17:37,790::task::993::Storage.TaskManager.Task::(_decref) Task=`dc5077c8-2a97-49bd-b359-0fd37ef4794f`::ref 0 aborting False >Thread-70000::DEBUG::2014-09-25 17:17:37,791::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70000::DEBUG::2014-09-25 17:17:37,791::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:38,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:38,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202981 s, 50.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:38,160::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:38,162::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70001::DEBUG::2014-09-25 17:17:38,164::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:40,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:40,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168749 s, 60.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:41,180::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:41,182::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70002::DEBUG::2014-09-25 17:17:41,184::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:42,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:42,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160307 s, 63.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:42,287::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:42,287::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:42,288::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:42,288::__init__::232::IOProcess::(_processLogs) (107685) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:42,289::__init__::232::IOProcess::(_processLogs) (107685) Queuing response >Thread-79::DEBUG::2014-09-25 17:17:42,289::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:17:42,298::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000295089 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:42,299::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:42,300::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:42,300::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:42,301::__init__::232::IOProcess::(_processLogs) (107686) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:42,302::__init__::232::IOProcess::(_processLogs) (107686) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:42,302::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:42,302::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:42,303::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:42,303::__init__::232::IOProcess::(_processLogs) (107687) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:42,304::__init__::232::IOProcess::(_processLogs) (107687) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:42,304::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:42,304::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:42,305::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:42,306::__init__::232::IOProcess::(_processLogs) (107688) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:42,306::__init__::232::IOProcess::(_processLogs) (107688) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:42,306::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:42,306::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:42,307::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:42,307::__init__::232::IOProcess::(_processLogs) (107689) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:42,308::__init__::232::IOProcess::(_processLogs) (107689) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:42,957::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:42,957::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:42,957::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:42,957::__init__::232::IOProcess::(_processLogs) (107690) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:42,958::__init__::232::IOProcess::(_processLogs) (107690) Queuing response >Thread-80::DEBUG::2014-09-25 17:17:42,960::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:17:42,969::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000430252 s, 879 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:42,970::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:42,970::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:42,971::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:42,972::__init__::232::IOProcess::(_processLogs) (107691) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:42,973::__init__::232::IOProcess::(_processLogs) (107691) Queuing response >Dummy-148::DEBUG::2014-09-25 17:17:44,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:44,202::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:44,203::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70003::DEBUG::2014-09-25 17:17:44,206::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:44,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170653 s, 60.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:17:46,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:46,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.22993 s, 4.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:47,223::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:47,224::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70004::DEBUG::2014-09-25 17:17:47,226::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:17:47,702::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:47,703::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:47,703::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:47,703::__init__::232::IOProcess::(_processLogs) (107692) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:47,705::__init__::232::IOProcess::(_processLogs) (107692) Queuing response >Thread-81::DEBUG::2014-09-25 17:17:47,706::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:17:47,722::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00486261 s, 76.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:47,723::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:47,724::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:47,724::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:47,724::__init__::232::IOProcess::(_processLogs) (107693) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:47,726::__init__::232::IOProcess::(_processLogs) (107693) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:47,820::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:47,821::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70005::DEBUG::2014-09-25 17:17:47,821::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70005::DEBUG::2014-09-25 17:17:47,823::task::595::Storage.TaskManager.Task::(_updateState) Task=`22856b5f-c4b1-45b4-a14b-7d3ed87c700f`::moving from state init -> state preparing >Thread-70005::INFO::2014-09-25 17:17:47,823::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70005::INFO::2014-09-25 17:17:47,824::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70005::DEBUG::2014-09-25 17:17:47,824::task::1191::Storage.TaskManager.Task::(prepare) Task=`22856b5f-c4b1-45b4-a14b-7d3ed87c700f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70005::DEBUG::2014-09-25 17:17:47,824::task::595::Storage.TaskManager.Task::(_updateState) Task=`22856b5f-c4b1-45b4-a14b-7d3ed87c700f`::moving from state preparing -> state finished >Thread-70005::DEBUG::2014-09-25 17:17:47,825::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70005::DEBUG::2014-09-25 17:17:47,825::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70005::DEBUG::2014-09-25 17:17:47,825::task::993::Storage.TaskManager.Task::(_decref) Task=`22856b5f-c4b1-45b4-a14b-7d3ed87c700f`::ref 0 aborting False >Thread-70005::DEBUG::2014-09-25 17:17:47,825::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70005::DEBUG::2014-09-25 17:17:47,826::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:47,840::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:47,841::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70006::DEBUG::2014-09-25 17:17:47,841::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70006::DEBUG::2014-09-25 17:17:47,843::task::595::Storage.TaskManager.Task::(_updateState) Task=`4e4616d4-cf3d-4e37-bcec-b7a3c0292a23`::moving from state init -> state preparing >Thread-70006::INFO::2014-09-25 17:17:47,843::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70006::DEBUG::2014-09-25 17:17:47,844::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`09cd10dd-d8d5-4cfb-af75-c6b15b353f89`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70006::DEBUG::2014-09-25 17:17:47,844::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70006::DEBUG::2014-09-25 17:17:47,844::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70006::DEBUG::2014-09-25 17:17:47,845::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`09cd10dd-d8d5-4cfb-af75-c6b15b353f89`::Granted request >Thread-70006::DEBUG::2014-09-25 17:17:47,846::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4e4616d4-cf3d-4e37-bcec-b7a3c0292a23`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70006::DEBUG::2014-09-25 17:17:47,847::task::993::Storage.TaskManager.Task::(_decref) Task=`4e4616d4-cf3d-4e37-bcec-b7a3c0292a23`::ref 1 aborting False >Thread-70006::INFO::2014-09-25 17:17:47,851::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70006::DEBUG::2014-09-25 17:17:47,851::task::1191::Storage.TaskManager.Task::(prepare) Task=`4e4616d4-cf3d-4e37-bcec-b7a3c0292a23`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70006::DEBUG::2014-09-25 17:17:47,851::task::595::Storage.TaskManager.Task::(_updateState) Task=`4e4616d4-cf3d-4e37-bcec-b7a3c0292a23`::moving from state preparing -> state finished >Thread-70006::DEBUG::2014-09-25 17:17:47,852::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70006::DEBUG::2014-09-25 17:17:47,852::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70006::DEBUG::2014-09-25 17:17:47,852::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70006::DEBUG::2014-09-25 17:17:47,852::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70006::DEBUG::2014-09-25 17:17:47,853::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70006::DEBUG::2014-09-25 17:17:47,853::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70006::DEBUG::2014-09-25 17:17:47,853::task::993::Storage.TaskManager.Task::(_decref) Task=`4e4616d4-cf3d-4e37-bcec-b7a3c0292a23`::ref 0 aborting False >Thread-70006::DEBUG::2014-09-25 17:17:47,854::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70006::DEBUG::2014-09-25 17:17:47,854::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:48,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:48,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204718 s, 50.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:50,240::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:50,242::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70007::DEBUG::2014-09-25 17:17:50,245::task::595::Storage.TaskManager.Task::(_updateState) Task=`7fcaeaa3-fb90-4580-b5c0-52ad0653355e`::moving from state init -> state preparing >Thread-70007::INFO::2014-09-25 17:17:50,245::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70007::INFO::2014-09-25 17:17:50,245::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000295089', 'lastCheck': '7.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000430252', 'lastCheck': '7.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00486261', 'lastCheck': '2.5', 'valid': True}} >Thread-70007::DEBUG::2014-09-25 17:17:50,246::task::1191::Storage.TaskManager.Task::(prepare) Task=`7fcaeaa3-fb90-4580-b5c0-52ad0653355e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000295089', 'lastCheck': '7.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000430252', 'lastCheck': '7.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00486261', 'lastCheck': '2.5', 'valid': True}} >Thread-70007::DEBUG::2014-09-25 17:17:50,246::task::595::Storage.TaskManager.Task::(_updateState) Task=`7fcaeaa3-fb90-4580-b5c0-52ad0653355e`::moving from state preparing -> state finished >Thread-70007::DEBUG::2014-09-25 17:17:50,246::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70007::DEBUG::2014-09-25 17:17:50,246::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70007::DEBUG::2014-09-25 17:17:50,247::task::993::Storage.TaskManager.Task::(_decref) Task=`7fcaeaa3-fb90-4580-b5c0-52ad0653355e`::ref 0 aborting False >Thread-70007::DEBUG::2014-09-25 17:17:50,249::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:50,271::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:50,272::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70008::DEBUG::2014-09-25 17:17:50,274::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:50,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:50,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166374 s, 61.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:52,309::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:52,309::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:52,310::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:52,311::__init__::232::IOProcess::(_processLogs) (107694) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 17:17:52,312::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:17:52,312::__init__::232::IOProcess::(_processLogs) (107694) Queuing response >Thread-79::DEBUG::2014-09-25 17:17:52,321::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000385233 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:52,322::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:52,323::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:52,323::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:52,323::__init__::232::IOProcess::(_processLogs) (107695) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:52,324::__init__::232::IOProcess::(_processLogs) (107695) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:52,325::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:52,325::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:52,326::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:52,327::__init__::232::IOProcess::(_processLogs) (107696) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:52,327::__init__::232::IOProcess::(_processLogs) (107696) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:52,327::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:52,328::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:52,328::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:52,329::__init__::232::IOProcess::(_processLogs) (107697) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:52,329::__init__::232::IOProcess::(_processLogs) (107697) Queuing response >Thread-29::DEBUG::2014-09-25 17:17:52,329::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:52,330::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:52,331::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:52,331::__init__::232::IOProcess::(_processLogs) (107698) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:17:52,331::__init__::232::IOProcess::(_processLogs) (107698) Queuing response >Dummy-148::DEBUG::2014-09-25 17:17:52,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:52,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151352 s, 67.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:52,974::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:52,974::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:52,974::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:52,974::__init__::232::IOProcess::(_processLogs) (107699) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:52,975::__init__::232::IOProcess::(_processLogs) (107699) Queuing response >Thread-80::DEBUG::2014-09-25 17:17:52,976::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:17:52,985::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000563372 s, 671 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:52,986::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:52,986::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:52,986::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:52,987::__init__::232::IOProcess::(_processLogs) (107700) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:52,987::__init__::232::IOProcess::(_processLogs) (107700) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:53,337::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:53,338::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70009::DEBUG::2014-09-25 17:17:53,340::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:54,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:54,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213253 s, 48.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:56,359::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:56,360::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70010::DEBUG::2014-09-25 17:17:56,362::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:56,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:56,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133473 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:57,728::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:57,728::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:57,729::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:57,729::__init__::232::IOProcess::(_processLogs) (107701) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:57,739::__init__::232::IOProcess::(_processLogs) (107701) Queuing response >Thread-81::DEBUG::2014-09-25 17:17:57,740::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:17:57,767::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00474863 s, 78.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:17:57,768::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:17:57,768::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:17:57,768::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:17:57,769::__init__::232::IOProcess::(_processLogs) (107702) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:17:57,776::__init__::232::IOProcess::(_processLogs) (107702) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:57,883::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70011::DEBUG::2014-09-25 17:17:57,885::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:17:57,885::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70011::DEBUG::2014-09-25 17:17:57,887::task::595::Storage.TaskManager.Task::(_updateState) Task=`10097f3d-258e-44f2-944a-3e52cbc8734a`::moving from state init -> state preparing >Thread-70011::INFO::2014-09-25 17:17:57,887::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70011::INFO::2014-09-25 17:17:57,888::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70011::DEBUG::2014-09-25 17:17:57,888::task::1191::Storage.TaskManager.Task::(prepare) Task=`10097f3d-258e-44f2-944a-3e52cbc8734a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70011::DEBUG::2014-09-25 17:17:57,888::task::595::Storage.TaskManager.Task::(_updateState) Task=`10097f3d-258e-44f2-944a-3e52cbc8734a`::moving from state preparing -> state finished >Thread-70011::DEBUG::2014-09-25 17:17:57,889::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70011::DEBUG::2014-09-25 17:17:57,889::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70011::DEBUG::2014-09-25 17:17:57,889::task::993::Storage.TaskManager.Task::(_decref) Task=`10097f3d-258e-44f2-944a-3e52cbc8734a`::ref 0 aborting False >Thread-70011::DEBUG::2014-09-25 17:17:57,890::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70011::DEBUG::2014-09-25 17:17:57,890::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:57,904::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:57,905::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70012::DEBUG::2014-09-25 17:17:57,905::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70012::DEBUG::2014-09-25 17:17:57,907::task::595::Storage.TaskManager.Task::(_updateState) Task=`7dbc7ee9-2e6f-4e07-a37a-5ed27f0146ff`::moving from state init -> state preparing >Thread-70012::INFO::2014-09-25 17:17:57,907::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70012::DEBUG::2014-09-25 17:17:57,908::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`65561848-2e3a-44a4-adb8-4a76f180c106`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70012::DEBUG::2014-09-25 17:17:57,908::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70012::DEBUG::2014-09-25 17:17:57,909::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70012::DEBUG::2014-09-25 17:17:57,909::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`65561848-2e3a-44a4-adb8-4a76f180c106`::Granted request >Thread-70012::DEBUG::2014-09-25 17:17:57,909::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7dbc7ee9-2e6f-4e07-a37a-5ed27f0146ff`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70012::DEBUG::2014-09-25 17:17:57,910::task::993::Storage.TaskManager.Task::(_decref) Task=`7dbc7ee9-2e6f-4e07-a37a-5ed27f0146ff`::ref 1 aborting False >Thread-70012::INFO::2014-09-25 17:17:57,914::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70012::DEBUG::2014-09-25 17:17:57,914::task::1191::Storage.TaskManager.Task::(prepare) Task=`7dbc7ee9-2e6f-4e07-a37a-5ed27f0146ff`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70012::DEBUG::2014-09-25 17:17:57,914::task::595::Storage.TaskManager.Task::(_updateState) Task=`7dbc7ee9-2e6f-4e07-a37a-5ed27f0146ff`::moving from state preparing -> state finished >Thread-70012::DEBUG::2014-09-25 17:17:57,915::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70012::DEBUG::2014-09-25 17:17:57,915::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70012::DEBUG::2014-09-25 17:17:57,915::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70012::DEBUG::2014-09-25 17:17:57,916::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70012::DEBUG::2014-09-25 17:17:57,916::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70012::DEBUG::2014-09-25 17:17:57,916::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70012::DEBUG::2014-09-25 17:17:57,916::task::993::Storage.TaskManager.Task::(_decref) Task=`7dbc7ee9-2e6f-4e07-a37a-5ed27f0146ff`::ref 0 aborting False >Thread-70012::DEBUG::2014-09-25 17:17:57,917::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70012::DEBUG::2014-09-25 17:17:57,917::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:17:58,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:17:58,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165097 s, 62.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:17:59,381::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:17:59,383::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70013::DEBUG::2014-09-25 17:17:59,385::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:00,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:00,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164844 s, 62.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:02,332::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:02,332::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:02,332::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:02,333::__init__::232::IOProcess::(_processLogs) (107703) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:02,333::__init__::232::IOProcess::(_processLogs) (107703) Queuing response >Thread-79::DEBUG::2014-09-25 17:18:02,334::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:18:02,343::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000521743 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:02,345::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:02,345::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:02,345::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:02,346::__init__::232::IOProcess::(_processLogs) (107704) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:02,346::__init__::232::IOProcess::(_processLogs) (107704) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:02,348::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:02,348::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:02,349::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:02,349::__init__::232::IOProcess::(_processLogs) (107705) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:02,349::__init__::232::IOProcess::(_processLogs) (107705) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:02,350::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:02,350::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:02,351::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:02,351::__init__::232::IOProcess::(_processLogs) (107706) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:02,351::__init__::232::IOProcess::(_processLogs) (107706) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:02,352::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:02,352::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:02,353::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:02,354::__init__::232::IOProcess::(_processLogs) (107707) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:02,354::__init__::232::IOProcess::(_processLogs) (107707) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:02,402::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:02,404::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70014::DEBUG::2014-09-25 17:18:02,406::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:18:02,989::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:02,990::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:02,990::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:02,990::__init__::232::IOProcess::(_processLogs) (107708) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:02,991::__init__::232::IOProcess::(_processLogs) (107708) Queuing response >Thread-80::DEBUG::2014-09-25 17:18:02,992::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:18:03,001::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000449334 s, 841 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:03,002::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:03,002::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:03,003::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:03,003::__init__::232::IOProcess::(_processLogs) (107709) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:03,004::__init__::232::IOProcess::(_processLogs) (107709) Queuing response >Dummy-148::DEBUG::2014-09-25 17:18:03,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:03,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134075 s, 76.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:18:05,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:05,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209546 s, 48.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:05,422::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:05,424::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70015::DEBUG::2014-09-25 17:18:05,426::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f9a5055-5966-40a8-977d-c8dcd49b274f`::moving from state init -> state preparing >Thread-70015::INFO::2014-09-25 17:18:05,427::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70015::INFO::2014-09-25 17:18:05,429::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000521743', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449334', 'lastCheck': '2.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00474863', 'lastCheck': '7.7', 'valid': True}} >Thread-70015::DEBUG::2014-09-25 17:18:05,430::task::1191::Storage.TaskManager.Task::(prepare) Task=`2f9a5055-5966-40a8-977d-c8dcd49b274f`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000521743', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449334', 'lastCheck': '2.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00474863', 'lastCheck': '7.7', 'valid': True}} >Thread-70015::DEBUG::2014-09-25 17:18:05,431::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f9a5055-5966-40a8-977d-c8dcd49b274f`::moving from state preparing -> state finished >Thread-70015::DEBUG::2014-09-25 17:18:05,433::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70015::DEBUG::2014-09-25 17:18:05,433::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70015::DEBUG::2014-09-25 17:18:05,434::task::993::Storage.TaskManager.Task::(_decref) Task=`2f9a5055-5966-40a8-977d-c8dcd49b274f`::ref 0 aborting False >Thread-70015::DEBUG::2014-09-25 17:18:05,441::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:05,464::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:05,465::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70016::DEBUG::2014-09-25 17:18:05,467::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:07,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:07,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133524 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:07,778::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:07,778::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:07,778::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:07,779::__init__::232::IOProcess::(_processLogs) (107710) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:07,782::__init__::232::IOProcess::(_processLogs) (107710) Queuing response >Thread-81::DEBUG::2014-09-25 17:18:07,783::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:18:07,804::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00863062 s, 43.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:07,805::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:07,806::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:07,806::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:07,806::__init__::232::IOProcess::(_processLogs) (107711) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:07,812::__init__::232::IOProcess::(_processLogs) (107711) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:07,945::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:07,946::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70017::DEBUG::2014-09-25 17:18:07,946::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70017::DEBUG::2014-09-25 17:18:07,948::task::595::Storage.TaskManager.Task::(_updateState) Task=`12a10eb3-cfc8-47fe-91e0-49cfd08fc02a`::moving from state init -> state preparing >Thread-70017::INFO::2014-09-25 17:18:07,948::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70017::INFO::2014-09-25 17:18:07,949::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70017::DEBUG::2014-09-25 17:18:07,949::task::1191::Storage.TaskManager.Task::(prepare) Task=`12a10eb3-cfc8-47fe-91e0-49cfd08fc02a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70017::DEBUG::2014-09-25 17:18:07,949::task::595::Storage.TaskManager.Task::(_updateState) Task=`12a10eb3-cfc8-47fe-91e0-49cfd08fc02a`::moving from state preparing -> state finished >Thread-70017::DEBUG::2014-09-25 17:18:07,950::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70017::DEBUG::2014-09-25 17:18:07,950::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70017::DEBUG::2014-09-25 17:18:07,950::task::993::Storage.TaskManager.Task::(_decref) Task=`12a10eb3-cfc8-47fe-91e0-49cfd08fc02a`::ref 0 aborting False >Thread-70017::DEBUG::2014-09-25 17:18:07,950::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70017::DEBUG::2014-09-25 17:18:07,951::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:07,966::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:07,967::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70018::DEBUG::2014-09-25 17:18:07,967::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70018::DEBUG::2014-09-25 17:18:07,969::task::595::Storage.TaskManager.Task::(_updateState) Task=`a439725e-bc2a-4ecd-b9bf-e82951a54a8c`::moving from state init -> state preparing >Thread-70018::INFO::2014-09-25 17:18:07,969::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70018::DEBUG::2014-09-25 17:18:07,970::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c548e963-31d9-4a55-bcd2-309e59451983`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70018::DEBUG::2014-09-25 17:18:07,970::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70018::DEBUG::2014-09-25 17:18:07,970::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70018::DEBUG::2014-09-25 17:18:07,971::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c548e963-31d9-4a55-bcd2-309e59451983`::Granted request >Thread-70018::DEBUG::2014-09-25 17:18:07,971::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a439725e-bc2a-4ecd-b9bf-e82951a54a8c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70018::DEBUG::2014-09-25 17:18:07,971::task::993::Storage.TaskManager.Task::(_decref) Task=`a439725e-bc2a-4ecd-b9bf-e82951a54a8c`::ref 1 aborting False >Thread-70018::INFO::2014-09-25 17:18:07,976::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70018::DEBUG::2014-09-25 17:18:07,976::task::1191::Storage.TaskManager.Task::(prepare) Task=`a439725e-bc2a-4ecd-b9bf-e82951a54a8c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70018::DEBUG::2014-09-25 17:18:07,976::task::595::Storage.TaskManager.Task::(_updateState) Task=`a439725e-bc2a-4ecd-b9bf-e82951a54a8c`::moving from state preparing -> state finished >Thread-70018::DEBUG::2014-09-25 17:18:07,976::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70018::DEBUG::2014-09-25 17:18:07,977::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70018::DEBUG::2014-09-25 17:18:07,977::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70018::DEBUG::2014-09-25 17:18:07,977::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70018::DEBUG::2014-09-25 17:18:07,978::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70018::DEBUG::2014-09-25 17:18:07,978::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70018::DEBUG::2014-09-25 17:18:07,978::task::993::Storage.TaskManager.Task::(_decref) Task=`a439725e-bc2a-4ecd-b9bf-e82951a54a8c`::ref 0 aborting False >Thread-70018::DEBUG::2014-09-25 17:18:07,979::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70018::DEBUG::2014-09-25 17:18:07,979::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:08,526::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:08,527::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70019::DEBUG::2014-09-25 17:18:08,529::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:09,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:09,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153694 s, 66.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:18:11,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:11,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0176699 s, 58.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:11,546::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:11,548::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70020::DEBUG::2014-09-25 17:18:11,550::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:18:12,354::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:12,354::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:12,355::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:12,355::__init__::232::IOProcess::(_processLogs) (107712) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:12,363::__init__::232::IOProcess::(_processLogs) (107712) Queuing response >Thread-79::DEBUG::2014-09-25 17:18:12,365::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:18:12,382::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00137957 s, 407 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:12,383::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:12,383::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:12,383::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:12,383::__init__::232::IOProcess::(_processLogs) (107713) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:12,385::__init__::232::IOProcess::(_processLogs) (107713) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:12,387::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:12,387::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:12,387::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:12,387::__init__::232::IOProcess::(_processLogs) (107714) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:12,388::__init__::232::IOProcess::(_processLogs) (107714) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:12,389::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:12,390::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:12,390::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:12,390::__init__::232::IOProcess::(_processLogs) (107715) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:12,391::__init__::232::IOProcess::(_processLogs) (107715) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:12,391::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:12,392::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:12,393::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:12,393::__init__::232::IOProcess::(_processLogs) (107716) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:12,393::__init__::232::IOProcess::(_processLogs) (107716) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:13,006::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:13,006::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:13,006::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:13,006::__init__::232::IOProcess::(_processLogs) (107717) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:13,007::__init__::232::IOProcess::(_processLogs) (107717) Queuing response >Thread-80::DEBUG::2014-09-25 17:18:13,008::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:18:13,016::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000414257 s, 912 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:13,017::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:13,018::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:13,018::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:13,018::__init__::232::IOProcess::(_processLogs) (107718) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:13,019::__init__::232::IOProcess::(_processLogs) (107718) Queuing response >Dummy-148::DEBUG::2014-09-25 17:18:13,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:13,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183711 s, 55.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:14,566::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:14,567::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70021::DEBUG::2014-09-25 17:18:14,569::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:15,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:15,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133697 s, 76.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:18:17,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:17,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172365 s, 59.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:17,587::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:17,588::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70022::DEBUG::2014-09-25 17:18:17,590::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:18:17,813::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:17,814::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:17,814::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:17,814::__init__::232::IOProcess::(_processLogs) (107719) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:17,816::__init__::232::IOProcess::(_processLogs) (107719) Queuing response >Thread-81::DEBUG::2014-09-25 17:18:17,817::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:18:17,839::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00441712 s, 84.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:17,840::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:17,841::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:17,841::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:17,841::__init__::232::IOProcess::(_processLogs) (107720) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:17,848::__init__::232::IOProcess::(_processLogs) (107720) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:18,007::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70023::DEBUG::2014-09-25 17:18:18,008::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:18:18,008::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70023::DEBUG::2014-09-25 17:18:18,011::task::595::Storage.TaskManager.Task::(_updateState) Task=`63aa0181-8403-49e2-b767-377ef94881d1`::moving from state init -> state preparing >Thread-70023::INFO::2014-09-25 17:18:18,011::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70023::INFO::2014-09-25 17:18:18,012::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70023::DEBUG::2014-09-25 17:18:18,012::task::1191::Storage.TaskManager.Task::(prepare) Task=`63aa0181-8403-49e2-b767-377ef94881d1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70023::DEBUG::2014-09-25 17:18:18,012::task::595::Storage.TaskManager.Task::(_updateState) Task=`63aa0181-8403-49e2-b767-377ef94881d1`::moving from state preparing -> state finished >Thread-70023::DEBUG::2014-09-25 17:18:18,012::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70023::DEBUG::2014-09-25 17:18:18,012::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70023::DEBUG::2014-09-25 17:18:18,013::task::993::Storage.TaskManager.Task::(_decref) Task=`63aa0181-8403-49e2-b767-377ef94881d1`::ref 0 aborting False >Thread-70023::DEBUG::2014-09-25 17:18:18,013::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70023::DEBUG::2014-09-25 17:18:18,013::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:18,028::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:18,029::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70024::DEBUG::2014-09-25 17:18:18,029::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70024::DEBUG::2014-09-25 17:18:18,030::task::595::Storage.TaskManager.Task::(_updateState) Task=`22a24a8b-43f5-46ab-8fc7-f38bbcd1ae5c`::moving from state init -> state preparing >Thread-70024::INFO::2014-09-25 17:18:18,031::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70024::DEBUG::2014-09-25 17:18:18,031::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`96839b37-fe1d-41a3-a57c-0a352dc46176`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70024::DEBUG::2014-09-25 17:18:18,032::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70024::DEBUG::2014-09-25 17:18:18,032::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70024::DEBUG::2014-09-25 17:18:18,032::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`96839b37-fe1d-41a3-a57c-0a352dc46176`::Granted request >Thread-70024::DEBUG::2014-09-25 17:18:18,033::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`22a24a8b-43f5-46ab-8fc7-f38bbcd1ae5c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70024::DEBUG::2014-09-25 17:18:18,033::task::993::Storage.TaskManager.Task::(_decref) Task=`22a24a8b-43f5-46ab-8fc7-f38bbcd1ae5c`::ref 1 aborting False >Thread-70024::INFO::2014-09-25 17:18:18,037::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70024::DEBUG::2014-09-25 17:18:18,038::task::1191::Storage.TaskManager.Task::(prepare) Task=`22a24a8b-43f5-46ab-8fc7-f38bbcd1ae5c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70024::DEBUG::2014-09-25 17:18:18,038::task::595::Storage.TaskManager.Task::(_updateState) Task=`22a24a8b-43f5-46ab-8fc7-f38bbcd1ae5c`::moving from state preparing -> state finished >Thread-70024::DEBUG::2014-09-25 17:18:18,038::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70024::DEBUG::2014-09-25 17:18:18,038::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70024::DEBUG::2014-09-25 17:18:18,039::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70024::DEBUG::2014-09-25 17:18:18,039::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70024::DEBUG::2014-09-25 17:18:18,039::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70024::DEBUG::2014-09-25 17:18:18,039::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70024::DEBUG::2014-09-25 17:18:18,040::task::993::Storage.TaskManager.Task::(_decref) Task=`22a24a8b-43f5-46ab-8fc7-f38bbcd1ae5c`::ref 0 aborting False >Thread-70024::DEBUG::2014-09-25 17:18:18,040::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70024::DEBUG::2014-09-25 17:18:18,041::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:19,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:19,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.022268 s, 46.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:20,605::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:20,607::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70025::DEBUG::2014-09-25 17:18:20,609::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a86ff30-17ce-4c34-972f-8232854e55b8`::moving from state init -> state preparing >Thread-70025::INFO::2014-09-25 17:18:20,610::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70025::INFO::2014-09-25 17:18:20,610::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00137957', 'lastCheck': '8.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000414257', 'lastCheck': '7.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00441712', 'lastCheck': '2.8', 'valid': True}} >Thread-70025::DEBUG::2014-09-25 17:18:20,610::task::1191::Storage.TaskManager.Task::(prepare) Task=`9a86ff30-17ce-4c34-972f-8232854e55b8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00137957', 'lastCheck': '8.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000414257', 'lastCheck': '7.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00441712', 'lastCheck': '2.8', 'valid': True}} >Thread-70025::DEBUG::2014-09-25 17:18:20,611::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a86ff30-17ce-4c34-972f-8232854e55b8`::moving from state preparing -> state finished >Thread-70025::DEBUG::2014-09-25 17:18:20,611::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70025::DEBUG::2014-09-25 17:18:20,611::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70025::DEBUG::2014-09-25 17:18:20,611::task::993::Storage.TaskManager.Task::(_decref) Task=`9a86ff30-17ce-4c34-972f-8232854e55b8`::ref 0 aborting False >Thread-70025::DEBUG::2014-09-25 17:18:20,614::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:20,634::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:20,635::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70026::DEBUG::2014-09-25 17:18:20,637::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:21,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:21,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165893 s, 61.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:22,393::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:22,394::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:22,394::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:22,394::__init__::232::IOProcess::(_processLogs) (107721) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:22,399::__init__::232::IOProcess::(_processLogs) (107721) Queuing response >Thread-79::DEBUG::2014-09-25 17:18:22,400::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:18:22,413::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00113213 s, 496 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:22,414::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:22,414::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:22,415::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:22,415::__init__::232::IOProcess::(_processLogs) (107722) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:22,418::__init__::232::IOProcess::(_processLogs) (107722) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:22,419::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:22,419::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:22,420::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:22,420::__init__::232::IOProcess::(_processLogs) (107723) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:22,420::__init__::232::IOProcess::(_processLogs) (107723) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:22,421::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:22,421::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:22,422::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:22,422::__init__::232::IOProcess::(_processLogs) (107724) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:22,422::__init__::232::IOProcess::(_processLogs) (107724) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:22,423::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:22,423::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:22,424::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:22,425::__init__::232::IOProcess::(_processLogs) (107725) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:22,425::__init__::232::IOProcess::(_processLogs) (107725) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:23,021::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:23,021::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:23,022::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:23,022::__init__::232::IOProcess::(_processLogs) (107726) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:23,022::__init__::232::IOProcess::(_processLogs) (107726) Queuing response >Thread-80::DEBUG::2014-09-25 17:18:23,023::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:18:23,032::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000578653 s, 653 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:23,033::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:23,033::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:23,034::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:23,034::__init__::232::IOProcess::(_processLogs) (107727) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:23,034::__init__::232::IOProcess::(_processLogs) (107727) Queuing response >Dummy-148::DEBUG::2014-09-25 17:18:23,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:23,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0140754 s, 72.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:23,695::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:23,697::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70027::DEBUG::2014-09-25 17:18:23,699::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:25,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:25,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159154 s, 64.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:26,717::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:26,719::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70028::DEBUG::2014-09-25 17:18:26,721::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:27,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:27,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202475 s, 50.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:27,850::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:27,850::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:27,850::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:27,850::__init__::232::IOProcess::(_processLogs) (107728) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:27,851::__init__::232::IOProcess::(_processLogs) (107728) Queuing response >Thread-81::DEBUG::2014-09-25 17:18:27,852::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:18:27,863::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00253215 s, 147 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:27,864::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:27,865::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:27,865::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:27,865::__init__::232::IOProcess::(_processLogs) (107729) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:27,868::__init__::232::IOProcess::(_processLogs) (107729) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:28,069::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:28,070::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70029::DEBUG::2014-09-25 17:18:28,070::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70029::DEBUG::2014-09-25 17:18:28,072::task::595::Storage.TaskManager.Task::(_updateState) Task=`14cea3fd-264e-4906-ad9f-21aa56ce6279`::moving from state init -> state preparing >Thread-70029::INFO::2014-09-25 17:18:28,073::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70029::INFO::2014-09-25 17:18:28,073::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70029::DEBUG::2014-09-25 17:18:28,073::task::1191::Storage.TaskManager.Task::(prepare) Task=`14cea3fd-264e-4906-ad9f-21aa56ce6279`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70029::DEBUG::2014-09-25 17:18:28,074::task::595::Storage.TaskManager.Task::(_updateState) Task=`14cea3fd-264e-4906-ad9f-21aa56ce6279`::moving from state preparing -> state finished >Thread-70029::DEBUG::2014-09-25 17:18:28,074::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70029::DEBUG::2014-09-25 17:18:28,074::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70029::DEBUG::2014-09-25 17:18:28,074::task::993::Storage.TaskManager.Task::(_decref) Task=`14cea3fd-264e-4906-ad9f-21aa56ce6279`::ref 0 aborting False >Thread-70029::DEBUG::2014-09-25 17:18:28,075::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70029::DEBUG::2014-09-25 17:18:28,075::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:28,088::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:28,090::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70030::DEBUG::2014-09-25 17:18:28,090::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70030::DEBUG::2014-09-25 17:18:28,091::task::595::Storage.TaskManager.Task::(_updateState) Task=`3bb91e79-1f9e-4e37-b1fa-38e7e6b1d726`::moving from state init -> state preparing >Thread-70030::INFO::2014-09-25 17:18:28,092::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70030::DEBUG::2014-09-25 17:18:28,092::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`29879be4-6e41-4fe8-8ed4-0a0c4390d40a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70030::DEBUG::2014-09-25 17:18:28,093::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70030::DEBUG::2014-09-25 17:18:28,093::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70030::DEBUG::2014-09-25 17:18:28,093::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`29879be4-6e41-4fe8-8ed4-0a0c4390d40a`::Granted request >Thread-70030::DEBUG::2014-09-25 17:18:28,094::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3bb91e79-1f9e-4e37-b1fa-38e7e6b1d726`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70030::DEBUG::2014-09-25 17:18:28,094::task::993::Storage.TaskManager.Task::(_decref) Task=`3bb91e79-1f9e-4e37-b1fa-38e7e6b1d726`::ref 1 aborting False >Thread-70030::INFO::2014-09-25 17:18:28,098::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70030::DEBUG::2014-09-25 17:18:28,099::task::1191::Storage.TaskManager.Task::(prepare) Task=`3bb91e79-1f9e-4e37-b1fa-38e7e6b1d726`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70030::DEBUG::2014-09-25 17:18:28,099::task::595::Storage.TaskManager.Task::(_updateState) Task=`3bb91e79-1f9e-4e37-b1fa-38e7e6b1d726`::moving from state preparing -> state finished >Thread-70030::DEBUG::2014-09-25 17:18:28,099::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70030::DEBUG::2014-09-25 17:18:28,099::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70030::DEBUG::2014-09-25 17:18:28,100::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70030::DEBUG::2014-09-25 17:18:28,100::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70030::DEBUG::2014-09-25 17:18:28,100::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70030::DEBUG::2014-09-25 17:18:28,100::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70030::DEBUG::2014-09-25 17:18:28,101::task::993::Storage.TaskManager.Task::(_decref) Task=`3bb91e79-1f9e-4e37-b1fa-38e7e6b1d726`::ref 0 aborting False >Thread-70030::DEBUG::2014-09-25 17:18:28,101::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70030::DEBUG::2014-09-25 17:18:28,102::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:29,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:29,739::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:29,741::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70031::DEBUG::2014-09-25 17:18:29,743::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:29,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.213464 s, 4.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:18:31,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:31,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163207 s, 62.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:32,425::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:32,425::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:32,426::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:32,426::__init__::232::IOProcess::(_processLogs) (107730) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:32,430::__init__::232::IOProcess::(_processLogs) (107730) Queuing response >Thread-79::DEBUG::2014-09-25 17:18:32,430::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:18:32,442::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00286111 s, 196 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:32,444::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:32,444::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:32,444::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:32,444::__init__::232::IOProcess::(_processLogs) (107731) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:32,445::__init__::232::IOProcess::(_processLogs) (107731) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:32,447::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:32,447::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:32,448::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:32,448::__init__::232::IOProcess::(_processLogs) (107732) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:32,448::__init__::232::IOProcess::(_processLogs) (107732) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:32,449::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:32,449::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:32,450::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:32,450::__init__::232::IOProcess::(_processLogs) (107733) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:32,450::__init__::232::IOProcess::(_processLogs) (107733) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:32,451::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:32,452::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:32,452::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:32,453::__init__::232::IOProcess::(_processLogs) (107734) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:32,453::__init__::232::IOProcess::(_processLogs) (107734) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:32,760::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:32,761::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70032::DEBUG::2014-09-25 17:18:32,763::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:18:33,036::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:33,036::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:33,036::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:33,036::__init__::232::IOProcess::(_processLogs) (107735) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:33,037::__init__::232::IOProcess::(_processLogs) (107735) Queuing response >Thread-80::DEBUG::2014-09-25 17:18:33,038::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:18:33,047::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000517026 s, 731 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:33,048::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:33,049::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:33,049::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:33,049::__init__::232::IOProcess::(_processLogs) (107736) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:33,050::__init__::232::IOProcess::(_processLogs) (107736) Queuing response >Dummy-148::DEBUG::2014-09-25 17:18:33,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:33,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134155 s, 76.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:35,778::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:35,780::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70033::DEBUG::2014-09-25 17:18:35,783::task::595::Storage.TaskManager.Task::(_updateState) Task=`244f588b-ddd2-45aa-b160-4cf7d101427b`::moving from state init -> state preparing >Thread-70033::INFO::2014-09-25 17:18:35,783::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70033::INFO::2014-09-25 17:18:35,784::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00286111', 'lastCheck': '3.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000517026', 'lastCheck': '2.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00253215', 'lastCheck': '7.9', 'valid': True}} >Thread-70033::DEBUG::2014-09-25 17:18:35,784::task::1191::Storage.TaskManager.Task::(prepare) Task=`244f588b-ddd2-45aa-b160-4cf7d101427b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00286111', 'lastCheck': '3.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000517026', 'lastCheck': '2.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00253215', 'lastCheck': '7.9', 'valid': True}} >Thread-70033::DEBUG::2014-09-25 17:18:35,784::task::595::Storage.TaskManager.Task::(_updateState) Task=`244f588b-ddd2-45aa-b160-4cf7d101427b`::moving from state preparing -> state finished >Thread-70033::DEBUG::2014-09-25 17:18:35,784::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70033::DEBUG::2014-09-25 17:18:35,784::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70033::DEBUG::2014-09-25 17:18:35,785::task::993::Storage.TaskManager.Task::(_decref) Task=`244f588b-ddd2-45aa-b160-4cf7d101427b`::ref 0 aborting False >Thread-70033::DEBUG::2014-09-25 17:18:35,787::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:35,809::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:35,810::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70034::DEBUG::2014-09-25 17:18:35,812::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:35,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:35,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017205 s, 59.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:37,870::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:37,870::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:37,870::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:37,871::__init__::232::IOProcess::(_processLogs) (107737) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:37,873::__init__::232::IOProcess::(_processLogs) (107737) Queuing response >Thread-81::DEBUG::2014-09-25 17:18:37,873::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:18:37,892::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00292811 s, 127 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:37,894::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:37,894::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:37,894::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:37,894::__init__::232::IOProcess::(_processLogs) (107738) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:37,898::__init__::232::IOProcess::(_processLogs) (107738) Queuing response >Dummy-148::DEBUG::2014-09-25 17:18:38,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:38,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174155 s, 58.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:38,131::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:38,133::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70035::DEBUG::2014-09-25 17:18:38,133::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70035::DEBUG::2014-09-25 17:18:38,135::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a1ebde7-7834-4942-aa60-c5201f2f8238`::moving from state init -> state preparing >Thread-70035::INFO::2014-09-25 17:18:38,135::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70035::INFO::2014-09-25 17:18:38,136::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70035::DEBUG::2014-09-25 17:18:38,136::task::1191::Storage.TaskManager.Task::(prepare) Task=`9a1ebde7-7834-4942-aa60-c5201f2f8238`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70035::DEBUG::2014-09-25 17:18:38,136::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a1ebde7-7834-4942-aa60-c5201f2f8238`::moving from state preparing -> state finished >Thread-70035::DEBUG::2014-09-25 17:18:38,136::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70035::DEBUG::2014-09-25 17:18:38,137::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70035::DEBUG::2014-09-25 17:18:38,137::task::993::Storage.TaskManager.Task::(_decref) Task=`9a1ebde7-7834-4942-aa60-c5201f2f8238`::ref 0 aborting False >Thread-70035::DEBUG::2014-09-25 17:18:38,137::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70035::DEBUG::2014-09-25 17:18:38,138::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:38,151::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70036::DEBUG::2014-09-25 17:18:38,153::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:18:38,153::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70036::DEBUG::2014-09-25 17:18:38,155::task::595::Storage.TaskManager.Task::(_updateState) Task=`894e523c-4ab5-47c5-8830-5c90677b8d84`::moving from state init -> state preparing >Thread-70036::INFO::2014-09-25 17:18:38,155::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70036::DEBUG::2014-09-25 17:18:38,156::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`86d46d68-349c-431b-961d-1b966b2f6c14`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70036::DEBUG::2014-09-25 17:18:38,156::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70036::DEBUG::2014-09-25 17:18:38,156::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70036::DEBUG::2014-09-25 17:18:38,157::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`86d46d68-349c-431b-961d-1b966b2f6c14`::Granted request >Thread-70036::DEBUG::2014-09-25 17:18:38,157::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`894e523c-4ab5-47c5-8830-5c90677b8d84`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70036::DEBUG::2014-09-25 17:18:38,157::task::993::Storage.TaskManager.Task::(_decref) Task=`894e523c-4ab5-47c5-8830-5c90677b8d84`::ref 1 aborting False >Thread-70036::INFO::2014-09-25 17:18:38,162::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70036::DEBUG::2014-09-25 17:18:38,162::task::1191::Storage.TaskManager.Task::(prepare) Task=`894e523c-4ab5-47c5-8830-5c90677b8d84`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70036::DEBUG::2014-09-25 17:18:38,162::task::595::Storage.TaskManager.Task::(_updateState) Task=`894e523c-4ab5-47c5-8830-5c90677b8d84`::moving from state preparing -> state finished >Thread-70036::DEBUG::2014-09-25 17:18:38,163::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70036::DEBUG::2014-09-25 17:18:38,163::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70036::DEBUG::2014-09-25 17:18:38,163::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70036::DEBUG::2014-09-25 17:18:38,163::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70036::DEBUG::2014-09-25 17:18:38,164::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70036::DEBUG::2014-09-25 17:18:38,164::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70036::DEBUG::2014-09-25 17:18:38,164::task::993::Storage.TaskManager.Task::(_decref) Task=`894e523c-4ab5-47c5-8830-5c90677b8d84`::ref 0 aborting False >Thread-70036::DEBUG::2014-09-25 17:18:38,165::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70036::DEBUG::2014-09-25 17:18:38,165::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:38,871::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:38,872::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70037::DEBUG::2014-09-25 17:18:38,874::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:40,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:40,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134007 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:41,891::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:41,893::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70038::DEBUG::2014-09-25 17:18:41,895::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:42,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:42,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153459 s, 66.7 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 17:18:42,452::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:18:42,456::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,457::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,458::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,458::__init__::232::IOProcess::(_processLogs) (107739) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:18:42,458::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,459::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,459::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,459::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,459::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,460::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,460::__init__::232::IOProcess::(_processLogs) (107740) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:18:42,460::__init__::232::IOProcess::(_processLogs) (107741) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:18:42,461::__init__::232::IOProcess::(_processLogs) (107740) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:42,461::__init__::232::IOProcess::(_processLogs) (107739) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:42,462::__init__::232::IOProcess::(_processLogs) (107741) Queuing response >Thread-79::DEBUG::2014-09-25 17:18:42,466::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:18:42,467::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,467::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,467::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,468::__init__::232::IOProcess::(_processLogs) (107742) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:18:42,494::__init__::232::IOProcess::(_processLogs) (107742) Queuing response >Thread-79::DEBUG::2014-09-25 17:18:42,495::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:18:42,495::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,496::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,496::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,496::__init__::232::IOProcess::(_processLogs) (107743) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:18:42,504::__init__::232::IOProcess::(_processLogs) (107743) Queuing response >Thread-79::DEBUG::2014-09-25 17:18:42,504::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:18:42,505::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,506::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,506::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,506::__init__::232::IOProcess::(_processLogs) (107744) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:42,507::__init__::232::IOProcess::(_processLogs) (107744) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:42,508::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,508::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,508::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,508::__init__::232::IOProcess::(_processLogs) (107745) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:18:42,515::__init__::232::IOProcess::(_processLogs) (107745) Queuing response >Thread-79::DEBUG::2014-09-25 17:18:42,515::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 17:18:42,516::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:18:42,516::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:18:42,517::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,517::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,518::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,518::__init__::232::IOProcess::(_processLogs) (107746) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:42,523::__init__::232::IOProcess::(_processLogs) (107746) Queuing response >Thread-79::DEBUG::2014-09-25 17:18:42,524::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:18:42,538::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00411378 s, 137 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:42,540::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,540::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,540::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,540::__init__::232::IOProcess::(_processLogs) (107747) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:42,543::__init__::232::IOProcess::(_processLogs) (107747) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:42,544::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,544::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,545::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,545::__init__::232::IOProcess::(_processLogs) (107748) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:42,547::__init__::232::IOProcess::(_processLogs) (107748) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:42,548::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,548::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,548::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,549::__init__::232::IOProcess::(_processLogs) (107749) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:42,553::__init__::232::IOProcess::(_processLogs) (107749) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:42,554::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:42,554::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:42,554::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:42,554::__init__::232::IOProcess::(_processLogs) (107750) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:42,557::__init__::232::IOProcess::(_processLogs) (107750) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:43,052::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:43,052::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:43,052::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:43,053::__init__::232::IOProcess::(_processLogs) (107751) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:43,053::__init__::232::IOProcess::(_processLogs) (107751) Queuing response >Thread-80::DEBUG::2014-09-25 17:18:43,055::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:18:43,064::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000465908 s, 811 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:43,065::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:43,065::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:43,066::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:43,067::__init__::232::IOProcess::(_processLogs) (107752) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:43,068::__init__::232::IOProcess::(_processLogs) (107752) Queuing response >Dummy-148::DEBUG::2014-09-25 17:18:44,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:44,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013229 s, 77.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:44,913::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:44,915::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70042::DEBUG::2014-09-25 17:18:44,917::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:46,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:46,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0240655 s, 42.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:47,900::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:47,900::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:47,900::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:47,900::__init__::232::IOProcess::(_processLogs) (107753) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:47,904::__init__::232::IOProcess::(_processLogs) (107753) Queuing response >Thread-81::DEBUG::2014-09-25 17:18:47,904::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:18:47,919::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00394264 s, 94.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:47,920::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:47,921::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:47,921::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:47,921::__init__::232::IOProcess::(_processLogs) (107754) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:47,922::__init__::232::IOProcess::(_processLogs) (107754) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:47,934::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:47,935::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70043::DEBUG::2014-09-25 17:18:47,937::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:48,195::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70044::DEBUG::2014-09-25 17:18:48,196::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:18:48,196::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70044::DEBUG::2014-09-25 17:18:48,198::task::595::Storage.TaskManager.Task::(_updateState) Task=`79c35df0-d672-476a-b10d-4723f59a30ee`::moving from state init -> state preparing >Thread-70044::INFO::2014-09-25 17:18:48,199::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70044::INFO::2014-09-25 17:18:48,199::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70044::DEBUG::2014-09-25 17:18:48,199::task::1191::Storage.TaskManager.Task::(prepare) Task=`79c35df0-d672-476a-b10d-4723f59a30ee`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70044::DEBUG::2014-09-25 17:18:48,200::task::595::Storage.TaskManager.Task::(_updateState) Task=`79c35df0-d672-476a-b10d-4723f59a30ee`::moving from state preparing -> state finished >Thread-70044::DEBUG::2014-09-25 17:18:48,200::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70044::DEBUG::2014-09-25 17:18:48,200::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70044::DEBUG::2014-09-25 17:18:48,200::task::993::Storage.TaskManager.Task::(_decref) Task=`79c35df0-d672-476a-b10d-4723f59a30ee`::ref 0 aborting False >Thread-70044::DEBUG::2014-09-25 17:18:48,201::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70044::DEBUG::2014-09-25 17:18:48,201::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:48,215::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:48,216::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70045::DEBUG::2014-09-25 17:18:48,216::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70045::DEBUG::2014-09-25 17:18:48,218::task::595::Storage.TaskManager.Task::(_updateState) Task=`14a25f5d-7d3c-4a62-a4e7-d7e04f26703d`::moving from state init -> state preparing >Thread-70045::INFO::2014-09-25 17:18:48,218::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70045::DEBUG::2014-09-25 17:18:48,219::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d5ccb4c5-59c5-4692-80dd-8439da9ab8fc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70045::DEBUG::2014-09-25 17:18:48,219::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70045::DEBUG::2014-09-25 17:18:48,220::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70045::DEBUG::2014-09-25 17:18:48,220::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d5ccb4c5-59c5-4692-80dd-8439da9ab8fc`::Granted request >Thread-70045::DEBUG::2014-09-25 17:18:48,220::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`14a25f5d-7d3c-4a62-a4e7-d7e04f26703d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70045::DEBUG::2014-09-25 17:18:48,221::task::993::Storage.TaskManager.Task::(_decref) Task=`14a25f5d-7d3c-4a62-a4e7-d7e04f26703d`::ref 1 aborting False >Thread-70045::INFO::2014-09-25 17:18:48,225::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70045::DEBUG::2014-09-25 17:18:48,225::task::1191::Storage.TaskManager.Task::(prepare) Task=`14a25f5d-7d3c-4a62-a4e7-d7e04f26703d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70045::DEBUG::2014-09-25 17:18:48,225::task::595::Storage.TaskManager.Task::(_updateState) Task=`14a25f5d-7d3c-4a62-a4e7-d7e04f26703d`::moving from state preparing -> state finished >Thread-70045::DEBUG::2014-09-25 17:18:48,226::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70045::DEBUG::2014-09-25 17:18:48,226::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70045::DEBUG::2014-09-25 17:18:48,226::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70045::DEBUG::2014-09-25 17:18:48,227::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70045::DEBUG::2014-09-25 17:18:48,227::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70045::DEBUG::2014-09-25 17:18:48,227::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70045::DEBUG::2014-09-25 17:18:48,227::task::993::Storage.TaskManager.Task::(_decref) Task=`14a25f5d-7d3c-4a62-a4e7-d7e04f26703d`::ref 0 aborting False >Thread-70045::DEBUG::2014-09-25 17:18:48,228::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70045::DEBUG::2014-09-25 17:18:48,228::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:48,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:48,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133661 s, 76.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:18:50,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:50,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168209 s, 60.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:50,952::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:50,953::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70046::DEBUG::2014-09-25 17:18:50,956::task::595::Storage.TaskManager.Task::(_updateState) Task=`b95069b4-efa5-4bc1-88a7-ca3266152bf6`::moving from state init -> state preparing >Thread-70046::INFO::2014-09-25 17:18:50,956::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70046::INFO::2014-09-25 17:18:50,957::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00411378', 'lastCheck': '8.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000465908', 'lastCheck': '7.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00394264', 'lastCheck': '3.0', 'valid': True}} >Thread-70046::DEBUG::2014-09-25 17:18:50,957::task::1191::Storage.TaskManager.Task::(prepare) Task=`b95069b4-efa5-4bc1-88a7-ca3266152bf6`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00411378', 'lastCheck': '8.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000465908', 'lastCheck': '7.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00394264', 'lastCheck': '3.0', 'valid': True}} >Thread-70046::DEBUG::2014-09-25 17:18:50,957::task::595::Storage.TaskManager.Task::(_updateState) Task=`b95069b4-efa5-4bc1-88a7-ca3266152bf6`::moving from state preparing -> state finished >Thread-70046::DEBUG::2014-09-25 17:18:50,958::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70046::DEBUG::2014-09-25 17:18:50,958::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70046::DEBUG::2014-09-25 17:18:50,958::task::993::Storage.TaskManager.Task::(_decref) Task=`b95069b4-efa5-4bc1-88a7-ca3266152bf6`::ref 0 aborting False >Thread-70046::DEBUG::2014-09-25 17:18:50,960::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:50,982::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:50,984::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70047::DEBUG::2014-09-25 17:18:50,986::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:52,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:52,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152634 s, 67.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:52,559::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:52,559::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:52,560::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:52,560::__init__::232::IOProcess::(_processLogs) (107755) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:52,565::__init__::232::IOProcess::(_processLogs) (107755) Queuing response >Thread-79::DEBUG::2014-09-25 17:18:52,566::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:18:52,579::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00347862 s, 162 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:52,580::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:52,580::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:52,580::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:52,581::__init__::232::IOProcess::(_processLogs) (107756) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:52,586::__init__::232::IOProcess::(_processLogs) (107756) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:52,588::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:52,588::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:52,589::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:52,589::__init__::232::IOProcess::(_processLogs) (107757) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:52,590::__init__::232::IOProcess::(_processLogs) (107757) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:52,590::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:52,590::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:52,591::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:52,591::__init__::232::IOProcess::(_processLogs) (107758) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:52,592::__init__::232::IOProcess::(_processLogs) (107758) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:52,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:52,593::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:52,593::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:52,594::__init__::232::IOProcess::(_processLogs) (107759) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:18:52,594::__init__::232::IOProcess::(_processLogs) (107759) Queuing response >Thread-29::DEBUG::2014-09-25 17:18:53,068::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:53,068::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:53,069::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:53,069::__init__::232::IOProcess::(_processLogs) (107760) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:53,069::__init__::232::IOProcess::(_processLogs) (107760) Queuing response >Thread-80::DEBUG::2014-09-25 17:18:53,070::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:18:53,079::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000499685 s, 756 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:53,080::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:53,080::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:53,081::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:53,081::__init__::232::IOProcess::(_processLogs) (107761) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:53,082::__init__::232::IOProcess::(_processLogs) (107761) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:54,050::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:54,052::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70048::DEBUG::2014-09-25 17:18:54,054::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:54,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:54,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138657 s, 73.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:18:56,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:56,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0255291 s, 40.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:57,071::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:57,073::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70049::DEBUG::2014-09-25 17:18:57,076::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:18:57,925::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:57,926::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:57,926::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:57,926::__init__::232::IOProcess::(_processLogs) (107762) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:57,932::__init__::232::IOProcess::(_processLogs) (107762) Queuing response >Thread-81::DEBUG::2014-09-25 17:18:57,932::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:18:57,951::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00296546 s, 125 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:18:57,952::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:18:57,953::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:18:57,953::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:18:57,953::__init__::232::IOProcess::(_processLogs) (107763) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:18:57,955::__init__::232::IOProcess::(_processLogs) (107763) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:58,258::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:18:58,260::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70050::DEBUG::2014-09-25 17:18:58,260::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70050::DEBUG::2014-09-25 17:18:58,262::task::595::Storage.TaskManager.Task::(_updateState) Task=`7090609d-6c14-4535-9602-fd8af7ed8f1c`::moving from state init -> state preparing >Thread-70050::INFO::2014-09-25 17:18:58,262::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70050::INFO::2014-09-25 17:18:58,263::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70050::DEBUG::2014-09-25 17:18:58,263::task::1191::Storage.TaskManager.Task::(prepare) Task=`7090609d-6c14-4535-9602-fd8af7ed8f1c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70050::DEBUG::2014-09-25 17:18:58,263::task::595::Storage.TaskManager.Task::(_updateState) Task=`7090609d-6c14-4535-9602-fd8af7ed8f1c`::moving from state preparing -> state finished >Thread-70050::DEBUG::2014-09-25 17:18:58,264::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70050::DEBUG::2014-09-25 17:18:58,264::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70050::DEBUG::2014-09-25 17:18:58,264::task::993::Storage.TaskManager.Task::(_decref) Task=`7090609d-6c14-4535-9602-fd8af7ed8f1c`::ref 0 aborting False >Thread-70050::DEBUG::2014-09-25 17:18:58,264::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70050::DEBUG::2014-09-25 17:18:58,265::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:18:58,282::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70051::DEBUG::2014-09-25 17:18:58,283::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:18:58,283::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70051::DEBUG::2014-09-25 17:18:58,285::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5ac87cf-87d4-4ab6-bb73-07bbb82e8ad8`::moving from state init -> state preparing >Thread-70051::INFO::2014-09-25 17:18:58,286::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70051::DEBUG::2014-09-25 17:18:58,286::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`628b81b5-c3ff-4eb4-b7fb-8840a29641aa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70051::DEBUG::2014-09-25 17:18:58,287::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70051::DEBUG::2014-09-25 17:18:58,287::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70051::DEBUG::2014-09-25 17:18:58,287::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`628b81b5-c3ff-4eb4-b7fb-8840a29641aa`::Granted request >Thread-70051::DEBUG::2014-09-25 17:18:58,288::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f5ac87cf-87d4-4ab6-bb73-07bbb82e8ad8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70051::DEBUG::2014-09-25 17:18:58,288::task::993::Storage.TaskManager.Task::(_decref) Task=`f5ac87cf-87d4-4ab6-bb73-07bbb82e8ad8`::ref 1 aborting False >Thread-70051::INFO::2014-09-25 17:18:58,292::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70051::DEBUG::2014-09-25 17:18:58,293::task::1191::Storage.TaskManager.Task::(prepare) Task=`f5ac87cf-87d4-4ab6-bb73-07bbb82e8ad8`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70051::DEBUG::2014-09-25 17:18:58,293::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5ac87cf-87d4-4ab6-bb73-07bbb82e8ad8`::moving from state preparing -> state finished >Thread-70051::DEBUG::2014-09-25 17:18:58,293::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70051::DEBUG::2014-09-25 17:18:58,293::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70051::DEBUG::2014-09-25 17:18:58,294::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70051::DEBUG::2014-09-25 17:18:58,294::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70051::DEBUG::2014-09-25 17:18:58,294::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70051::DEBUG::2014-09-25 17:18:58,294::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70051::DEBUG::2014-09-25 17:18:58,295::task::993::Storage.TaskManager.Task::(_decref) Task=`f5ac87cf-87d4-4ab6-bb73-07bbb82e8ad8`::ref 0 aborting False >Thread-70051::DEBUG::2014-09-25 17:18:58,295::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70051::DEBUG::2014-09-25 17:18:58,296::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:18:58,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:18:58,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154337 s, 66.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:00,091::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:00,093::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70052::DEBUG::2014-09-25 17:19:00,095::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:00,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:00,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0245398 s, 41.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:02,594::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:02,594::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:02,595::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:02,595::__init__::232::IOProcess::(_processLogs) (107764) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:02,597::__init__::232::IOProcess::(_processLogs) (107764) Queuing response >Thread-79::DEBUG::2014-09-25 17:19:02,597::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:19:02,614::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00454547 s, 124 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:02,616::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:02,616::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:02,616::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:02,616::__init__::232::IOProcess::(_processLogs) (107765) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:02,620::__init__::232::IOProcess::(_processLogs) (107765) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:02,621::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:02,622::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:02,622::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:02,623::__init__::232::IOProcess::(_processLogs) (107766) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:02,623::__init__::232::IOProcess::(_processLogs) (107766) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:02,623::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:02,624::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:02,624::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:02,625::__init__::232::IOProcess::(_processLogs) (107767) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:02,625::__init__::232::IOProcess::(_processLogs) (107767) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:02,625::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:02,626::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:02,626::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:02,626::__init__::232::IOProcess::(_processLogs) (107768) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:02,626::__init__::232::IOProcess::(_processLogs) (107768) Queuing response >Dummy-148::DEBUG::2014-09-25 17:19:02,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:02,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0140662 s, 72.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:03,084::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:03,084::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:03,084::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:03,085::__init__::232::IOProcess::(_processLogs) (107769) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:03,085::__init__::232::IOProcess::(_processLogs) (107769) Queuing response >Thread-80::DEBUG::2014-09-25 17:19:03,086::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:19:03,095::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00042385 s, 892 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:03,096::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:03,097::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:03,097::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:03,097::__init__::232::IOProcess::(_processLogs) (107770) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:03,098::__init__::232::IOProcess::(_processLogs) (107770) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:03,117::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:03,118::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70053::DEBUG::2014-09-25 17:19:03,120::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:04,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:04,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164998 s, 62.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:06,137::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:06,138::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70054::DEBUG::2014-09-25 17:19:06,141::task::595::Storage.TaskManager.Task::(_updateState) Task=`024f5b82-a791-4257-9f53-fbcdadd1a508`::moving from state init -> state preparing >Thread-70054::INFO::2014-09-25 17:19:06,141::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70054::INFO::2014-09-25 17:19:06,142::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00454547', 'lastCheck': '3.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00042385', 'lastCheck': '3.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00296546', 'lastCheck': '8.2', 'valid': True}} >Thread-70054::DEBUG::2014-09-25 17:19:06,142::task::1191::Storage.TaskManager.Task::(prepare) Task=`024f5b82-a791-4257-9f53-fbcdadd1a508`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00454547', 'lastCheck': '3.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00042385', 'lastCheck': '3.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00296546', 'lastCheck': '8.2', 'valid': True}} >Thread-70054::DEBUG::2014-09-25 17:19:06,142::task::595::Storage.TaskManager.Task::(_updateState) Task=`024f5b82-a791-4257-9f53-fbcdadd1a508`::moving from state preparing -> state finished >Thread-70054::DEBUG::2014-09-25 17:19:06,142::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70054::DEBUG::2014-09-25 17:19:06,143::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70054::DEBUG::2014-09-25 17:19:06,143::task::993::Storage.TaskManager.Task::(_decref) Task=`024f5b82-a791-4257-9f53-fbcdadd1a508`::ref 0 aborting False >Thread-70054::DEBUG::2014-09-25 17:19:06,145::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:06,171::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:06,172::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70055::DEBUG::2014-09-25 17:19:06,174::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:06,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:06,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189172 s, 54.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:07,957::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:07,957::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:07,957::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:07,958::__init__::232::IOProcess::(_processLogs) (107771) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:07,958::__init__::232::IOProcess::(_processLogs) (107771) Queuing response >Thread-81::DEBUG::2014-09-25 17:19:07,959::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:19:07,977::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00866722 s, 42.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:07,979::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:07,979::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:07,979::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:07,979::__init__::232::IOProcess::(_processLogs) (107772) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:07,986::__init__::232::IOProcess::(_processLogs) (107772) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:08,337::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70056::DEBUG::2014-09-25 17:19:08,338::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:19:08,338::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70056::DEBUG::2014-09-25 17:19:08,340::task::595::Storage.TaskManager.Task::(_updateState) Task=`92f8206f-314f-4b48-9025-513965a4e7e5`::moving from state init -> state preparing >Thread-70056::INFO::2014-09-25 17:19:08,341::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70056::INFO::2014-09-25 17:19:08,341::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70056::DEBUG::2014-09-25 17:19:08,341::task::1191::Storage.TaskManager.Task::(prepare) Task=`92f8206f-314f-4b48-9025-513965a4e7e5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70056::DEBUG::2014-09-25 17:19:08,342::task::595::Storage.TaskManager.Task::(_updateState) Task=`92f8206f-314f-4b48-9025-513965a4e7e5`::moving from state preparing -> state finished >Thread-70056::DEBUG::2014-09-25 17:19:08,342::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70056::DEBUG::2014-09-25 17:19:08,342::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70056::DEBUG::2014-09-25 17:19:08,342::task::993::Storage.TaskManager.Task::(_decref) Task=`92f8206f-314f-4b48-9025-513965a4e7e5`::ref 0 aborting False >Thread-70056::DEBUG::2014-09-25 17:19:08,343::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70056::DEBUG::2014-09-25 17:19:08,343::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:08,357::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:08,358::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70057::DEBUG::2014-09-25 17:19:08,358::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70057::DEBUG::2014-09-25 17:19:08,360::task::595::Storage.TaskManager.Task::(_updateState) Task=`14217d4f-784d-41d2-ac96-25ba2c232041`::moving from state init -> state preparing >Thread-70057::INFO::2014-09-25 17:19:08,360::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70057::DEBUG::2014-09-25 17:19:08,361::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7174b3ee-6cf3-45e0-a330-e5606085f4e9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70057::DEBUG::2014-09-25 17:19:08,361::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70057::DEBUG::2014-09-25 17:19:08,362::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70057::DEBUG::2014-09-25 17:19:08,362::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7174b3ee-6cf3-45e0-a330-e5606085f4e9`::Granted request >Thread-70057::DEBUG::2014-09-25 17:19:08,363::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`14217d4f-784d-41d2-ac96-25ba2c232041`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70057::DEBUG::2014-09-25 17:19:08,363::task::993::Storage.TaskManager.Task::(_decref) Task=`14217d4f-784d-41d2-ac96-25ba2c232041`::ref 1 aborting False >Thread-70057::INFO::2014-09-25 17:19:08,367::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70057::DEBUG::2014-09-25 17:19:08,367::task::1191::Storage.TaskManager.Task::(prepare) Task=`14217d4f-784d-41d2-ac96-25ba2c232041`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70057::DEBUG::2014-09-25 17:19:08,368::task::595::Storage.TaskManager.Task::(_updateState) Task=`14217d4f-784d-41d2-ac96-25ba2c232041`::moving from state preparing -> state finished >Thread-70057::DEBUG::2014-09-25 17:19:08,368::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70057::DEBUG::2014-09-25 17:19:08,368::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70057::DEBUG::2014-09-25 17:19:08,368::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70057::DEBUG::2014-09-25 17:19:08,369::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70057::DEBUG::2014-09-25 17:19:08,369::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70057::DEBUG::2014-09-25 17:19:08,369::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70057::DEBUG::2014-09-25 17:19:08,370::task::993::Storage.TaskManager.Task::(_decref) Task=`14217d4f-784d-41d2-ac96-25ba2c232041`::ref 0 aborting False >Thread-70057::DEBUG::2014-09-25 17:19:08,370::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70057::DEBUG::2014-09-25 17:19:08,371::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:08,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:08,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177348 s, 57.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:09,236::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:09,237::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70058::DEBUG::2014-09-25 17:19:09,239::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:10,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:10,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184428 s, 55.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:12,258::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:12,260::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70059::DEBUG::2014-09-25 17:19:12,261::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:19:12,628::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:12,628::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:12,628::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:12,628::__init__::232::IOProcess::(_processLogs) (107773) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:12,641::__init__::232::IOProcess::(_processLogs) (107773) Queuing response >Thread-79::DEBUG::2014-09-25 17:19:12,642::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:19:12,659::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00611075 s, 92.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:12,660::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:12,660::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:12,660::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:12,661::__init__::232::IOProcess::(_processLogs) (107774) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:12,667::__init__::232::IOProcess::(_processLogs) (107774) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:12,668::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:12,669::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:12,670::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:12,670::__init__::232::IOProcess::(_processLogs) (107775) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:12,670::__init__::232::IOProcess::(_processLogs) (107775) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:12,671::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:12,671::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:12,671::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:12,671::__init__::232::IOProcess::(_processLogs) (107776) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:12,672::__init__::232::IOProcess::(_processLogs) (107776) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:12,672::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:12,673::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:12,673::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:12,673::__init__::232::IOProcess::(_processLogs) (107777) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:12,673::__init__::232::IOProcess::(_processLogs) (107777) Queuing response >Dummy-148::DEBUG::2014-09-25 17:19:13,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:13,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0220568 s, 46.4 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 17:19:13,099::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:19:13,103::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:13,104::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:13,104::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:13,105::__init__::232::IOProcess::(_processLogs) (107778) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:19:13,105::__init__::232::IOProcess::(_processLogs) (107778) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:13,106::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:13,106::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:13,107::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:13,107::__init__::232::IOProcess::(_processLogs) (107779) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:19:13,108::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:13,108::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:13,109::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:13,109::__init__::232::IOProcess::(_processLogs) (107780) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:19:13,109::__init__::232::IOProcess::(_processLogs) (107780) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:13,109::__init__::232::IOProcess::(_processLogs) (107779) Queuing response >Thread-80::DEBUG::2014-09-25 17:19:13,113::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:19:13,114::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:13,114::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:13,114::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:13,115::__init__::232::IOProcess::(_processLogs) (107781) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:19:13,119::__init__::232::IOProcess::(_processLogs) (107781) Queuing response >Thread-80::DEBUG::2014-09-25 17:19:13,119::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:19:13,120::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:13,120::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:13,120::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:13,121::__init__::232::IOProcess::(_processLogs) (107782) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:19:13,121::__init__::232::IOProcess::(_processLogs) (107782) Queuing response >Thread-80::DEBUG::2014-09-25 17:19:13,121::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:19:13,123::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:13,124::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:13,124::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:13,124::__init__::232::IOProcess::(_processLogs) (107783) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:13,125::__init__::232::IOProcess::(_processLogs) (107783) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:13,125::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:13,125::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:13,126::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:13,126::__init__::232::IOProcess::(_processLogs) (107784) Got request for method 'glob' >Thread-80::DEBUG::2014-09-25 17:19:13,126::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:19:13,126::__init__::232::IOProcess::(_processLogs) (107784) Queuing response >Thread-80::INFO::2014-09-25 17:19:13,127::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:19:13,128::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:19:13,129::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:13,129::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:13,129::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:13,129::__init__::232::IOProcess::(_processLogs) (107785) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:13,130::__init__::232::IOProcess::(_processLogs) (107785) Queuing response >Thread-80::DEBUG::2014-09-25 17:19:13,131::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:19:13,140::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000427535 s, 884 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:13,141::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:13,141::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:13,141::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:13,142::__init__::232::IOProcess::(_processLogs) (107786) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:13,142::__init__::232::IOProcess::(_processLogs) (107786) Queuing response >Dummy-148::DEBUG::2014-09-25 17:19:15,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:15,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0219134 s, 46.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:15,277::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:15,278::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70063::DEBUG::2014-09-25 17:19:15,281::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:17,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:17,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.270231 s, 3.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:17,988::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:17,988::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:17,988::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:17,988::__init__::232::IOProcess::(_processLogs) (107787) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:17,991::__init__::232::IOProcess::(_processLogs) (107787) Queuing response >Thread-81::DEBUG::2014-09-25 17:19:17,992::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:19:18,001::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000355199 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:18,002::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:18,002::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:18,003::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:18,003::__init__::232::IOProcess::(_processLogs) (107788) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:18,003::__init__::232::IOProcess::(_processLogs) (107788) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:18,298::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:18,299::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70064::DEBUG::2014-09-25 17:19:18,301::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:18,402::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:18,403::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70065::DEBUG::2014-09-25 17:19:18,403::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70065::DEBUG::2014-09-25 17:19:18,405::task::595::Storage.TaskManager.Task::(_updateState) Task=`784fed5c-d53d-462b-ba58-854aa4ebcf0e`::moving from state init -> state preparing >Thread-70065::INFO::2014-09-25 17:19:18,405::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70065::INFO::2014-09-25 17:19:18,406::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70065::DEBUG::2014-09-25 17:19:18,406::task::1191::Storage.TaskManager.Task::(prepare) Task=`784fed5c-d53d-462b-ba58-854aa4ebcf0e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70065::DEBUG::2014-09-25 17:19:18,406::task::595::Storage.TaskManager.Task::(_updateState) Task=`784fed5c-d53d-462b-ba58-854aa4ebcf0e`::moving from state preparing -> state finished >Thread-70065::DEBUG::2014-09-25 17:19:18,407::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70065::DEBUG::2014-09-25 17:19:18,407::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70065::DEBUG::2014-09-25 17:19:18,407::task::993::Storage.TaskManager.Task::(_decref) Task=`784fed5c-d53d-462b-ba58-854aa4ebcf0e`::ref 0 aborting False >Thread-70065::DEBUG::2014-09-25 17:19:18,407::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70065::DEBUG::2014-09-25 17:19:18,408::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:18,421::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:18,422::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70066::DEBUG::2014-09-25 17:19:18,422::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70066::DEBUG::2014-09-25 17:19:18,424::task::595::Storage.TaskManager.Task::(_updateState) Task=`824e921e-4493-46e2-9d5a-86c9818e955e`::moving from state init -> state preparing >Thread-70066::INFO::2014-09-25 17:19:18,424::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70066::DEBUG::2014-09-25 17:19:18,425::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`030ca403-1339-4500-b046-a70662cc51dc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70066::DEBUG::2014-09-25 17:19:18,425::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70066::DEBUG::2014-09-25 17:19:18,426::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70066::DEBUG::2014-09-25 17:19:18,426::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`030ca403-1339-4500-b046-a70662cc51dc`::Granted request >Thread-70066::DEBUG::2014-09-25 17:19:18,427::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`824e921e-4493-46e2-9d5a-86c9818e955e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70066::DEBUG::2014-09-25 17:19:18,427::task::993::Storage.TaskManager.Task::(_decref) Task=`824e921e-4493-46e2-9d5a-86c9818e955e`::ref 1 aborting False >Thread-70066::INFO::2014-09-25 17:19:18,431::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70066::DEBUG::2014-09-25 17:19:18,431::task::1191::Storage.TaskManager.Task::(prepare) Task=`824e921e-4493-46e2-9d5a-86c9818e955e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70066::DEBUG::2014-09-25 17:19:18,432::task::595::Storage.TaskManager.Task::(_updateState) Task=`824e921e-4493-46e2-9d5a-86c9818e955e`::moving from state preparing -> state finished >Thread-70066::DEBUG::2014-09-25 17:19:18,432::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70066::DEBUG::2014-09-25 17:19:18,432::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70066::DEBUG::2014-09-25 17:19:18,432::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70066::DEBUG::2014-09-25 17:19:18,433::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70066::DEBUG::2014-09-25 17:19:18,433::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70066::DEBUG::2014-09-25 17:19:18,433::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70066::DEBUG::2014-09-25 17:19:18,434::task::993::Storage.TaskManager.Task::(_decref) Task=`824e921e-4493-46e2-9d5a-86c9818e955e`::ref 0 aborting False >Thread-70066::DEBUG::2014-09-25 17:19:18,434::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70066::DEBUG::2014-09-25 17:19:18,435::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:19,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:19,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.271077 s, 3.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:21,315::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:21,317::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70067::DEBUG::2014-09-25 17:19:21,320::task::595::Storage.TaskManager.Task::(_updateState) Task=`a67a7652-4c7b-455b-933f-4591dab05dbb`::moving from state init -> state preparing >Thread-70067::INFO::2014-09-25 17:19:21,320::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70067::INFO::2014-09-25 17:19:21,320::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00611075', 'lastCheck': '8.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000427535', 'lastCheck': '8.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000355199', 'lastCheck': '3.3', 'valid': True}} >Thread-70067::DEBUG::2014-09-25 17:19:21,321::task::1191::Storage.TaskManager.Task::(prepare) Task=`a67a7652-4c7b-455b-933f-4591dab05dbb`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00611075', 'lastCheck': '8.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000427535', 'lastCheck': '8.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000355199', 'lastCheck': '3.3', 'valid': True}} >Thread-70067::DEBUG::2014-09-25 17:19:21,321::task::595::Storage.TaskManager.Task::(_updateState) Task=`a67a7652-4c7b-455b-933f-4591dab05dbb`::moving from state preparing -> state finished >Thread-70067::DEBUG::2014-09-25 17:19:21,321::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70067::DEBUG::2014-09-25 17:19:21,321::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70067::DEBUG::2014-09-25 17:19:21,322::task::993::Storage.TaskManager.Task::(_decref) Task=`a67a7652-4c7b-455b-933f-4591dab05dbb`::ref 0 aborting False >Thread-70067::DEBUG::2014-09-25 17:19:21,324::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:21,342::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:21,343::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70068::DEBUG::2014-09-25 17:19:21,345::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:21,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:21,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137416 s, 74.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:22,675::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:22,675::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:22,675::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:22,676::__init__::232::IOProcess::(_processLogs) (107789) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:22,677::__init__::232::IOProcess::(_processLogs) (107789) Queuing response >Thread-79::DEBUG::2014-09-25 17:19:22,678::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:19:22,702::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.008053 s, 69.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:22,703::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:22,703::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:22,704::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:22,704::__init__::232::IOProcess::(_processLogs) (107790) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:22,706::__init__::232::IOProcess::(_processLogs) (107790) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:22,708::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:22,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:22,708::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:22,709::__init__::232::IOProcess::(_processLogs) (107791) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:22,709::__init__::232::IOProcess::(_processLogs) (107791) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:22,710::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:22,710::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:22,710::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:22,710::__init__::232::IOProcess::(_processLogs) (107792) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:22,711::__init__::232::IOProcess::(_processLogs) (107792) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:22,712::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:22,712::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:22,712::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:22,712::__init__::232::IOProcess::(_processLogs) (107793) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:22,713::__init__::232::IOProcess::(_processLogs) (107793) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:23,144::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:23,145::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:23,145::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:23,145::__init__::232::IOProcess::(_processLogs) (107794) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:23,146::__init__::232::IOProcess::(_processLogs) (107794) Queuing response >Thread-80::DEBUG::2014-09-25 17:19:23,147::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:19:23,155::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000545778 s, 693 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:23,157::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:23,157::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:23,158::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:23,159::__init__::232::IOProcess::(_processLogs) (107795) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:23,159::__init__::232::IOProcess::(_processLogs) (107795) Queuing response >Dummy-148::DEBUG::2014-09-25 17:19:23,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:23,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177188 s, 57.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:24,393::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:24,394::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70069::DEBUG::2014-09-25 17:19:24,396::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:25,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:25,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145297 s, 70.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:27,414::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:27,415::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70070::DEBUG::2014-09-25 17:19:27,417::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:27,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:27,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174864 s, 58.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:28,006::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:28,006::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:28,007::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:28,007::__init__::232::IOProcess::(_processLogs) (107796) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:28,008::__init__::232::IOProcess::(_processLogs) (107796) Queuing response >Thread-81::DEBUG::2014-09-25 17:19:28,008::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:19:28,023::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00363788 s, 102 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:28,024::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:28,024::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:28,024::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:28,025::__init__::232::IOProcess::(_processLogs) (107797) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:28,025::__init__::232::IOProcess::(_processLogs) (107797) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:28,463::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70071::DEBUG::2014-09-25 17:19:28,464::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:19:28,464::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70071::DEBUG::2014-09-25 17:19:28,467::task::595::Storage.TaskManager.Task::(_updateState) Task=`58b1e815-8841-424b-b051-9d4cf28e0e3f`::moving from state init -> state preparing >Thread-70071::INFO::2014-09-25 17:19:28,467::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70071::INFO::2014-09-25 17:19:28,468::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70071::DEBUG::2014-09-25 17:19:28,468::task::1191::Storage.TaskManager.Task::(prepare) Task=`58b1e815-8841-424b-b051-9d4cf28e0e3f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70071::DEBUG::2014-09-25 17:19:28,468::task::595::Storage.TaskManager.Task::(_updateState) Task=`58b1e815-8841-424b-b051-9d4cf28e0e3f`::moving from state preparing -> state finished >Thread-70071::DEBUG::2014-09-25 17:19:28,468::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70071::DEBUG::2014-09-25 17:19:28,469::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70071::DEBUG::2014-09-25 17:19:28,469::task::993::Storage.TaskManager.Task::(_decref) Task=`58b1e815-8841-424b-b051-9d4cf28e0e3f`::ref 0 aborting False >Thread-70071::DEBUG::2014-09-25 17:19:28,469::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70071::DEBUG::2014-09-25 17:19:28,470::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:28,482::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:28,484::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70072::DEBUG::2014-09-25 17:19:28,484::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70072::DEBUG::2014-09-25 17:19:28,485::task::595::Storage.TaskManager.Task::(_updateState) Task=`b5f79c53-bfbd-44d5-bad3-738b4b54c814`::moving from state init -> state preparing >Thread-70072::INFO::2014-09-25 17:19:28,486::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70072::DEBUG::2014-09-25 17:19:28,486::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ac91b0a9-4b40-40b8-a034-1cd82e89a422`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70072::DEBUG::2014-09-25 17:19:28,487::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70072::DEBUG::2014-09-25 17:19:28,487::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70072::DEBUG::2014-09-25 17:19:28,487::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ac91b0a9-4b40-40b8-a034-1cd82e89a422`::Granted request >Thread-70072::DEBUG::2014-09-25 17:19:28,488::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b5f79c53-bfbd-44d5-bad3-738b4b54c814`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70072::DEBUG::2014-09-25 17:19:28,488::task::993::Storage.TaskManager.Task::(_decref) Task=`b5f79c53-bfbd-44d5-bad3-738b4b54c814`::ref 1 aborting False >Thread-70072::INFO::2014-09-25 17:19:28,492::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70072::DEBUG::2014-09-25 17:19:28,493::task::1191::Storage.TaskManager.Task::(prepare) Task=`b5f79c53-bfbd-44d5-bad3-738b4b54c814`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70072::DEBUG::2014-09-25 17:19:28,493::task::595::Storage.TaskManager.Task::(_updateState) Task=`b5f79c53-bfbd-44d5-bad3-738b4b54c814`::moving from state preparing -> state finished >Thread-70072::DEBUG::2014-09-25 17:19:28,493::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70072::DEBUG::2014-09-25 17:19:28,493::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70072::DEBUG::2014-09-25 17:19:28,494::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70072::DEBUG::2014-09-25 17:19:28,494::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70072::DEBUG::2014-09-25 17:19:28,494::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70072::DEBUG::2014-09-25 17:19:28,495::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70072::DEBUG::2014-09-25 17:19:28,495::task::993::Storage.TaskManager.Task::(_decref) Task=`b5f79c53-bfbd-44d5-bad3-738b4b54c814`::ref 0 aborting False >Thread-70072::DEBUG::2014-09-25 17:19:28,495::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70072::DEBUG::2014-09-25 17:19:28,496::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:29,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:29,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149322 s, 68.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:30,438::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:30,439::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70073::DEBUG::2014-09-25 17:19:30,441::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:31,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:32,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0140729 s, 72.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:32,714::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:32,714::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:32,715::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:32,715::__init__::232::IOProcess::(_processLogs) (107798) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:32,716::__init__::232::IOProcess::(_processLogs) (107798) Queuing response >Thread-79::DEBUG::2014-09-25 17:19:32,716::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:19:32,729::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00217097 s, 259 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:32,730::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:32,730::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:32,730::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:32,731::__init__::232::IOProcess::(_processLogs) (107799) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:32,733::__init__::232::IOProcess::(_processLogs) (107799) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:32,734::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:32,735::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:32,735::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:32,735::__init__::232::IOProcess::(_processLogs) (107800) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:32,735::__init__::232::IOProcess::(_processLogs) (107800) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:32,736::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:32,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:32,738::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:32,738::__init__::232::IOProcess::(_processLogs) (107801) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:32,738::__init__::232::IOProcess::(_processLogs) (107801) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:32,739::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:32,739::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:32,740::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:32,740::__init__::232::IOProcess::(_processLogs) (107802) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:32,741::__init__::232::IOProcess::(_processLogs) (107802) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:33,160::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:33,160::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:33,160::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:33,160::__init__::232::IOProcess::(_processLogs) (107803) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:33,161::__init__::232::IOProcess::(_processLogs) (107803) Queuing response >Thread-80::DEBUG::2014-09-25 17:19:33,162::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:19:33,171::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00040581 s, 931 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:33,172::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:33,172::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:33,172::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:33,173::__init__::232::IOProcess::(_processLogs) (107804) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:33,173::__init__::232::IOProcess::(_processLogs) (107804) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:33,461::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:33,463::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70074::DEBUG::2014-09-25 17:19:33,465::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:34,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:34,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135405 s, 75.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:19:36,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:36,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169718 s, 60.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:36,478::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:36,479::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70075::DEBUG::2014-09-25 17:19:36,482::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c87d865-2d5a-4a43-b7bd-5680bc0d845f`::moving from state init -> state preparing >Thread-70075::INFO::2014-09-25 17:19:36,483::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70075::INFO::2014-09-25 17:19:36,483::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00217097', 'lastCheck': '3.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00040581', 'lastCheck': '3.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00363788', 'lastCheck': '8.5', 'valid': True}} >Thread-70075::DEBUG::2014-09-25 17:19:36,483::task::1191::Storage.TaskManager.Task::(prepare) Task=`6c87d865-2d5a-4a43-b7bd-5680bc0d845f`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00217097', 'lastCheck': '3.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00040581', 'lastCheck': '3.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00363788', 'lastCheck': '8.5', 'valid': True}} >Thread-70075::DEBUG::2014-09-25 17:19:36,484::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c87d865-2d5a-4a43-b7bd-5680bc0d845f`::moving from state preparing -> state finished >Thread-70075::DEBUG::2014-09-25 17:19:36,484::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70075::DEBUG::2014-09-25 17:19:36,484::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70075::DEBUG::2014-09-25 17:19:36,484::task::993::Storage.TaskManager.Task::(_decref) Task=`6c87d865-2d5a-4a43-b7bd-5680bc0d845f`::ref 0 aborting False >Thread-70075::DEBUG::2014-09-25 17:19:36,487::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:36,503::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:36,504::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70076::DEBUG::2014-09-25 17:19:36,506::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:19:38,027::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:38,027::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:38,028::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:38,028::__init__::232::IOProcess::(_processLogs) (107805) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:38,031::__init__::232::IOProcess::(_processLogs) (107805) Queuing response >Thread-81::DEBUG::2014-09-25 17:19:38,032::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:19:38,048::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00529338 s, 70.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:38,049::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:38,049::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:38,050::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:38,050::__init__::232::IOProcess::(_processLogs) (107806) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:38,054::__init__::232::IOProcess::(_processLogs) (107806) Queuing response >Dummy-148::DEBUG::2014-09-25 17:19:38,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:38,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.02288 s, 44.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:38,523::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70077::DEBUG::2014-09-25 17:19:38,525::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:19:38,525::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70077::DEBUG::2014-09-25 17:19:38,527::task::595::Storage.TaskManager.Task::(_updateState) Task=`7013cb9d-c70e-4af7-90b9-68e2badbbf23`::moving from state init -> state preparing >Thread-70077::INFO::2014-09-25 17:19:38,528::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70077::INFO::2014-09-25 17:19:38,529::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70077::DEBUG::2014-09-25 17:19:38,530::task::1191::Storage.TaskManager.Task::(prepare) Task=`7013cb9d-c70e-4af7-90b9-68e2badbbf23`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70077::DEBUG::2014-09-25 17:19:38,530::task::595::Storage.TaskManager.Task::(_updateState) Task=`7013cb9d-c70e-4af7-90b9-68e2badbbf23`::moving from state preparing -> state finished >Thread-70077::DEBUG::2014-09-25 17:19:38,530::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70077::DEBUG::2014-09-25 17:19:38,530::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70077::DEBUG::2014-09-25 17:19:38,530::task::993::Storage.TaskManager.Task::(_decref) Task=`7013cb9d-c70e-4af7-90b9-68e2badbbf23`::ref 0 aborting False >Thread-70077::DEBUG::2014-09-25 17:19:38,531::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70077::DEBUG::2014-09-25 17:19:38,531::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:38,545::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:38,546::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70078::DEBUG::2014-09-25 17:19:38,546::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70078::DEBUG::2014-09-25 17:19:38,548::task::595::Storage.TaskManager.Task::(_updateState) Task=`77f86730-9a0e-4c30-826e-09e2c6c53ec2`::moving from state init -> state preparing >Thread-70078::INFO::2014-09-25 17:19:38,548::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70078::DEBUG::2014-09-25 17:19:38,549::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1ca1e13a-f4c2-4f2d-8a35-f749192aa7fa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70078::DEBUG::2014-09-25 17:19:38,549::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70078::DEBUG::2014-09-25 17:19:38,549::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70078::DEBUG::2014-09-25 17:19:38,550::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1ca1e13a-f4c2-4f2d-8a35-f749192aa7fa`::Granted request >Thread-70078::DEBUG::2014-09-25 17:19:38,550::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`77f86730-9a0e-4c30-826e-09e2c6c53ec2`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70078::DEBUG::2014-09-25 17:19:38,550::task::993::Storage.TaskManager.Task::(_decref) Task=`77f86730-9a0e-4c30-826e-09e2c6c53ec2`::ref 1 aborting False >Thread-70078::INFO::2014-09-25 17:19:38,555::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70078::DEBUG::2014-09-25 17:19:38,555::task::1191::Storage.TaskManager.Task::(prepare) Task=`77f86730-9a0e-4c30-826e-09e2c6c53ec2`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70078::DEBUG::2014-09-25 17:19:38,555::task::595::Storage.TaskManager.Task::(_updateState) Task=`77f86730-9a0e-4c30-826e-09e2c6c53ec2`::moving from state preparing -> state finished >Thread-70078::DEBUG::2014-09-25 17:19:38,556::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70078::DEBUG::2014-09-25 17:19:38,556::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70078::DEBUG::2014-09-25 17:19:38,556::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70078::DEBUG::2014-09-25 17:19:38,556::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70078::DEBUG::2014-09-25 17:19:38,557::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70078::DEBUG::2014-09-25 17:19:38,557::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70078::DEBUG::2014-09-25 17:19:38,557::task::993::Storage.TaskManager.Task::(_decref) Task=`77f86730-9a0e-4c30-826e-09e2c6c53ec2`::ref 0 aborting False >Thread-70078::DEBUG::2014-09-25 17:19:38,558::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70078::DEBUG::2014-09-25 17:19:38,558::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:39,557::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:39,558::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70079::DEBUG::2014-09-25 17:19:39,560::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:40,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:40,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193384 s, 53.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:19:42,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:42,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191678 s, 53.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:42,577::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:42,579::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70080::DEBUG::2014-09-25 17:19:42,581::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:19:42,741::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:42,741::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:42,741::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:42,742::__init__::232::IOProcess::(_processLogs) (107807) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:42,745::__init__::232::IOProcess::(_processLogs) (107807) Queuing response >Thread-79::DEBUG::2014-09-25 17:19:42,746::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:19:42,759::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.002161 s, 260 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:42,760::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:42,760::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:42,760::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:42,761::__init__::232::IOProcess::(_processLogs) (107808) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:42,762::__init__::232::IOProcess::(_processLogs) (107808) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:42,763::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:42,763::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:42,763::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:42,764::__init__::232::IOProcess::(_processLogs) (107809) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:42,764::__init__::232::IOProcess::(_processLogs) (107809) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:42,765::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:42,765::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:42,766::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:42,766::__init__::232::IOProcess::(_processLogs) (107810) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:42,771::__init__::232::IOProcess::(_processLogs) (107810) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:42,772::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:42,773::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:42,773::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:42,773::__init__::232::IOProcess::(_processLogs) (107811) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:42,775::__init__::232::IOProcess::(_processLogs) (107811) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:43,175::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:43,176::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:43,176::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:43,176::__init__::232::IOProcess::(_processLogs) (107812) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:43,177::__init__::232::IOProcess::(_processLogs) (107812) Queuing response >Thread-80::DEBUG::2014-09-25 17:19:43,177::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:19:43,187::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000561652 s, 673 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:43,188::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:43,189::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:43,189::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:43,189::__init__::232::IOProcess::(_processLogs) (107813) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:43,190::__init__::232::IOProcess::(_processLogs) (107813) Queuing response >Dummy-148::DEBUG::2014-09-25 17:19:44,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:44,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153669 s, 66.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:45,598::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:45,600::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70081::DEBUG::2014-09-25 17:19:45,602::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:46,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:46,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0229697 s, 44.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:48,056::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:48,056::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:48,056::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:48,057::__init__::232::IOProcess::(_processLogs) (107814) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:48,060::__init__::232::IOProcess::(_processLogs) (107814) Queuing response >Thread-81::DEBUG::2014-09-25 17:19:48,061::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:19:48,075::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00462733 s, 80.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:48,077::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:48,077::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:48,077::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:48,077::__init__::232::IOProcess::(_processLogs) (107815) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:48,079::__init__::232::IOProcess::(_processLogs) (107815) Queuing response >Dummy-148::DEBUG::2014-09-25 17:19:48,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:48,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136489 s, 75.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:48,594::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:48,597::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70082::DEBUG::2014-09-25 17:19:48,598::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70082::DEBUG::2014-09-25 17:19:48,602::task::595::Storage.TaskManager.Task::(_updateState) Task=`62641f97-7a63-400f-916b-7dba02a5a7d4`::moving from state init -> state preparing >Thread-70082::INFO::2014-09-25 17:19:48,603::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70082::INFO::2014-09-25 17:19:48,605::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70082::DEBUG::2014-09-25 17:19:48,606::task::1191::Storage.TaskManager.Task::(prepare) Task=`62641f97-7a63-400f-916b-7dba02a5a7d4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70082::DEBUG::2014-09-25 17:19:48,607::task::595::Storage.TaskManager.Task::(_updateState) Task=`62641f97-7a63-400f-916b-7dba02a5a7d4`::moving from state preparing -> state finished >Thread-70082::DEBUG::2014-09-25 17:19:48,608::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70082::DEBUG::2014-09-25 17:19:48,609::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70082::DEBUG::2014-09-25 17:19:48,610::task::993::Storage.TaskManager.Task::(_decref) Task=`62641f97-7a63-400f-916b-7dba02a5a7d4`::ref 0 aborting False >Thread-70082::DEBUG::2014-09-25 17:19:48,611::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70082::DEBUG::2014-09-25 17:19:48,612::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:48,616::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:48,618::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70083::DEBUG::2014-09-25 17:19:48,622::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:48,626::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:48,628::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70084::DEBUG::2014-09-25 17:19:48,628::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70084::DEBUG::2014-09-25 17:19:48,632::task::595::Storage.TaskManager.Task::(_updateState) Task=`79c44fd6-ac55-4f9f-94b9-0d41a92620f2`::moving from state init -> state preparing >Thread-70084::INFO::2014-09-25 17:19:48,633::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70084::DEBUG::2014-09-25 17:19:48,635::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b138aaf6-6a09-44b6-9175-62084cce0de0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70084::DEBUG::2014-09-25 17:19:48,635::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70084::DEBUG::2014-09-25 17:19:48,637::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70084::DEBUG::2014-09-25 17:19:48,638::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b138aaf6-6a09-44b6-9175-62084cce0de0`::Granted request >Thread-70084::DEBUG::2014-09-25 17:19:48,640::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`79c44fd6-ac55-4f9f-94b9-0d41a92620f2`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70084::DEBUG::2014-09-25 17:19:48,641::task::993::Storage.TaskManager.Task::(_decref) Task=`79c44fd6-ac55-4f9f-94b9-0d41a92620f2`::ref 1 aborting False >Thread-70084::INFO::2014-09-25 17:19:48,655::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70084::DEBUG::2014-09-25 17:19:48,656::task::1191::Storage.TaskManager.Task::(prepare) Task=`79c44fd6-ac55-4f9f-94b9-0d41a92620f2`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70084::DEBUG::2014-09-25 17:19:48,657::task::595::Storage.TaskManager.Task::(_updateState) Task=`79c44fd6-ac55-4f9f-94b9-0d41a92620f2`::moving from state preparing -> state finished >Thread-70084::DEBUG::2014-09-25 17:19:48,658::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70084::DEBUG::2014-09-25 17:19:48,659::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70084::DEBUG::2014-09-25 17:19:48,661::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70084::DEBUG::2014-09-25 17:19:48,662::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70084::DEBUG::2014-09-25 17:19:48,662::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70084::DEBUG::2014-09-25 17:19:48,663::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70084::DEBUG::2014-09-25 17:19:48,665::task::993::Storage.TaskManager.Task::(_decref) Task=`79c44fd6-ac55-4f9f-94b9-0d41a92620f2`::ref 0 aborting False >Thread-70084::DEBUG::2014-09-25 17:19:48,666::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70084::DEBUG::2014-09-25 17:19:48,668::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:50,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:50,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162964 s, 62.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:51,634::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:51,636::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70085::DEBUG::2014-09-25 17:19:51,639::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc424bb1-0fb6-4541-98e4-a0acdfc80192`::moving from state init -> state preparing >Thread-70085::INFO::2014-09-25 17:19:51,639::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70085::INFO::2014-09-25 17:19:51,640::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.002161', 'lastCheck': '8.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000561652', 'lastCheck': '8.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00462733', 'lastCheck': '3.6', 'valid': True}} >Thread-70085::DEBUG::2014-09-25 17:19:51,640::task::1191::Storage.TaskManager.Task::(prepare) Task=`fc424bb1-0fb6-4541-98e4-a0acdfc80192`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.002161', 'lastCheck': '8.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000561652', 'lastCheck': '8.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00462733', 'lastCheck': '3.6', 'valid': True}} >Thread-70085::DEBUG::2014-09-25 17:19:51,640::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc424bb1-0fb6-4541-98e4-a0acdfc80192`::moving from state preparing -> state finished >Thread-70085::DEBUG::2014-09-25 17:19:51,641::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70085::DEBUG::2014-09-25 17:19:51,641::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70085::DEBUG::2014-09-25 17:19:51,641::task::993::Storage.TaskManager.Task::(_decref) Task=`fc424bb1-0fb6-4541-98e4-a0acdfc80192`::ref 0 aborting False >Thread-70085::DEBUG::2014-09-25 17:19:51,644::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:51,658::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:51,659::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70086::DEBUG::2014-09-25 17:19:51,661::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:52,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:52,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148917 s, 68.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:52,777::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:52,777::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:52,777::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:52,777::__init__::232::IOProcess::(_processLogs) (107816) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:52,778::__init__::232::IOProcess::(_processLogs) (107816) Queuing response >Thread-79::DEBUG::2014-09-25 17:19:52,779::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:19:52,793::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00509697 s, 110 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:52,794::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:52,794::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:52,795::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:52,795::__init__::232::IOProcess::(_processLogs) (107817) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:52,797::__init__::232::IOProcess::(_processLogs) (107817) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:52,799::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:52,799::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:52,800::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:52,800::__init__::232::IOProcess::(_processLogs) (107818) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:52,800::__init__::232::IOProcess::(_processLogs) (107818) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:52,801::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:52,801::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:52,802::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:52,802::__init__::232::IOProcess::(_processLogs) (107819) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:52,802::__init__::232::IOProcess::(_processLogs) (107819) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:52,803::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:52,803::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:52,803::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:52,804::__init__::232::IOProcess::(_processLogs) (107820) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:19:52,804::__init__::232::IOProcess::(_processLogs) (107820) Queuing response >Thread-29::DEBUG::2014-09-25 17:19:53,192::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:53,192::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:53,192::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:53,193::__init__::232::IOProcess::(_processLogs) (107821) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:53,193::__init__::232::IOProcess::(_processLogs) (107821) Queuing response >Thread-80::DEBUG::2014-09-25 17:19:53,194::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:19:53,203::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000433833 s, 871 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:53,204::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:53,204::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:53,204::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:53,204::__init__::232::IOProcess::(_processLogs) (107822) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:53,205::__init__::232::IOProcess::(_processLogs) (107822) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:54,744::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:54,745::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70087::DEBUG::2014-09-25 17:19:54,748::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:54,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:54,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013472 s, 76.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:19:56,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:56,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015071 s, 67.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:57,761::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:57,763::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70088::DEBUG::2014-09-25 17:19:57,765::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:19:58,081::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:58,082::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:58,082::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:58,082::__init__::232::IOProcess::(_processLogs) (107823) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:58,100::__init__::232::IOProcess::(_processLogs) (107823) Queuing response >Thread-81::DEBUG::2014-09-25 17:19:58,101::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:19:58,174::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0379886 s, 9.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:19:58,175::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:19:58,176::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:19:58,176::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:19:58,176::__init__::232::IOProcess::(_processLogs) (107824) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:19:58,182::__init__::232::IOProcess::(_processLogs) (107824) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:58,690::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70089::DEBUG::2014-09-25 17:19:58,691::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:19:58,691::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70089::DEBUG::2014-09-25 17:19:58,694::task::595::Storage.TaskManager.Task::(_updateState) Task=`da2b6d64-d9a9-4a36-9d23-071a3586b017`::moving from state init -> state preparing >Thread-70089::INFO::2014-09-25 17:19:58,694::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70089::INFO::2014-09-25 17:19:58,694::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70089::DEBUG::2014-09-25 17:19:58,695::task::1191::Storage.TaskManager.Task::(prepare) Task=`da2b6d64-d9a9-4a36-9d23-071a3586b017`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70089::DEBUG::2014-09-25 17:19:58,695::task::595::Storage.TaskManager.Task::(_updateState) Task=`da2b6d64-d9a9-4a36-9d23-071a3586b017`::moving from state preparing -> state finished >Thread-70089::DEBUG::2014-09-25 17:19:58,695::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70089::DEBUG::2014-09-25 17:19:58,695::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70089::DEBUG::2014-09-25 17:19:58,696::task::993::Storage.TaskManager.Task::(_decref) Task=`da2b6d64-d9a9-4a36-9d23-071a3586b017`::ref 0 aborting False >Thread-70089::DEBUG::2014-09-25 17:19:58,696::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70089::DEBUG::2014-09-25 17:19:58,696::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:19:58,709::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:19:58,710::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70090::DEBUG::2014-09-25 17:19:58,710::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70090::DEBUG::2014-09-25 17:19:58,712::task::595::Storage.TaskManager.Task::(_updateState) Task=`56d07bc3-5704-41ad-9d5c-8df7a00df7db`::moving from state init -> state preparing >Thread-70090::INFO::2014-09-25 17:19:58,712::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70090::DEBUG::2014-09-25 17:19:58,713::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2769df1d-300d-4ad8-a29c-f57d3411dc22`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70090::DEBUG::2014-09-25 17:19:58,713::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70090::DEBUG::2014-09-25 17:19:58,714::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70090::DEBUG::2014-09-25 17:19:58,714::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2769df1d-300d-4ad8-a29c-f57d3411dc22`::Granted request >Thread-70090::DEBUG::2014-09-25 17:19:58,714::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`56d07bc3-5704-41ad-9d5c-8df7a00df7db`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70090::DEBUG::2014-09-25 17:19:58,715::task::993::Storage.TaskManager.Task::(_decref) Task=`56d07bc3-5704-41ad-9d5c-8df7a00df7db`::ref 1 aborting False >Thread-70090::INFO::2014-09-25 17:19:58,719::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70090::DEBUG::2014-09-25 17:19:58,720::task::1191::Storage.TaskManager.Task::(prepare) Task=`56d07bc3-5704-41ad-9d5c-8df7a00df7db`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70090::DEBUG::2014-09-25 17:19:58,720::task::595::Storage.TaskManager.Task::(_updateState) Task=`56d07bc3-5704-41ad-9d5c-8df7a00df7db`::moving from state preparing -> state finished >Thread-70090::DEBUG::2014-09-25 17:19:58,720::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70090::DEBUG::2014-09-25 17:19:58,720::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70090::DEBUG::2014-09-25 17:19:58,721::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70090::DEBUG::2014-09-25 17:19:58,721::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70090::DEBUG::2014-09-25 17:19:58,721::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70090::DEBUG::2014-09-25 17:19:58,721::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70090::DEBUG::2014-09-25 17:19:58,722::task::993::Storage.TaskManager.Task::(_decref) Task=`56d07bc3-5704-41ad-9d5c-8df7a00df7db`::ref 0 aborting False >Thread-70090::DEBUG::2014-09-25 17:19:58,722::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70090::DEBUG::2014-09-25 17:19:58,723::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:19:58,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:19:58,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131735 s, 77.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:00,780::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:00,782::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70091::DEBUG::2014-09-25 17:20:00,784::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:00,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:00,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141535 s, 72.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:02,805::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:02,806::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:02,806::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:02,806::__init__::232::IOProcess::(_processLogs) (107825) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:02,815::__init__::232::IOProcess::(_processLogs) (107825) Queuing response >Thread-79::DEBUG::2014-09-25 17:20:02,816::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:20:02,835::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0036067 s, 156 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:02,836::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:02,836::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:02,837::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:02,837::__init__::232::IOProcess::(_processLogs) (107826) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:02,840::__init__::232::IOProcess::(_processLogs) (107826) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:02,841::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:02,841::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:02,842::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:02,843::__init__::232::IOProcess::(_processLogs) (107827) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:02,843::__init__::232::IOProcess::(_processLogs) (107827) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:02,843::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:02,844::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:02,844::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:02,845::__init__::232::IOProcess::(_processLogs) (107828) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:02,845::__init__::232::IOProcess::(_processLogs) (107828) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:02,845::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:02,846::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:02,847::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:02,847::__init__::232::IOProcess::(_processLogs) (107829) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:02,847::__init__::232::IOProcess::(_processLogs) (107829) Queuing response >Dummy-148::DEBUG::2014-09-25 17:20:02,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:02,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207919 s, 49.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:03,207::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:03,208::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:03,208::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:03,208::__init__::232::IOProcess::(_processLogs) (107830) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:03,209::__init__::232::IOProcess::(_processLogs) (107830) Queuing response >Thread-80::DEBUG::2014-09-25 17:20:03,210::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:20:03,219::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00051779 s, 730 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:03,220::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:03,220::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:03,220::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:03,221::__init__::232::IOProcess::(_processLogs) (107831) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:03,221::__init__::232::IOProcess::(_processLogs) (107831) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:03,797::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:03,799::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70092::DEBUG::2014-09-25 17:20:03,801::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:05,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:05,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185569 s, 55.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:06,812::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:06,813::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70093::DEBUG::2014-09-25 17:20:06,816::task::595::Storage.TaskManager.Task::(_updateState) Task=`25039407-2617-4c75-8c2e-88699a141dd8`::moving from state init -> state preparing >Thread-70093::INFO::2014-09-25 17:20:06,816::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70093::INFO::2014-09-25 17:20:06,817::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0036067', 'lastCheck': '4.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00051779', 'lastCheck': '3.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0379886', 'lastCheck': '8.6', 'valid': True}} >Thread-70093::DEBUG::2014-09-25 17:20:06,817::task::1191::Storage.TaskManager.Task::(prepare) Task=`25039407-2617-4c75-8c2e-88699a141dd8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0036067', 'lastCheck': '4.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00051779', 'lastCheck': '3.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0379886', 'lastCheck': '8.6', 'valid': True}} >Thread-70093::DEBUG::2014-09-25 17:20:06,817::task::595::Storage.TaskManager.Task::(_updateState) Task=`25039407-2617-4c75-8c2e-88699a141dd8`::moving from state preparing -> state finished >Thread-70093::DEBUG::2014-09-25 17:20:06,818::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70093::DEBUG::2014-09-25 17:20:06,818::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70093::DEBUG::2014-09-25 17:20:06,818::task::993::Storage.TaskManager.Task::(_decref) Task=`25039407-2617-4c75-8c2e-88699a141dd8`::ref 0 aborting False >Thread-70093::DEBUG::2014-09-25 17:20:06,820::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:06,840::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:06,841::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70094::DEBUG::2014-09-25 17:20:06,844::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:07,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:07,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0269529 s, 38.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:08,185::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:08,185::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:08,185::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:08,185::__init__::232::IOProcess::(_processLogs) (107832) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:08,190::__init__::232::IOProcess::(_processLogs) (107832) Queuing response >Thread-81::DEBUG::2014-09-25 17:20:08,191::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:20:08,205::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00464877 s, 79.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:08,206::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:08,207::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:08,207::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:08,207::__init__::232::IOProcess::(_processLogs) (107833) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:08,209::__init__::232::IOProcess::(_processLogs) (107833) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:08,752::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70095::DEBUG::2014-09-25 17:20:08,753::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:20:08,753::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70095::DEBUG::2014-09-25 17:20:08,755::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6cd18ff-cee7-4bd1-904a-7ba3da39eb5d`::moving from state init -> state preparing >Thread-70095::INFO::2014-09-25 17:20:08,756::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70095::INFO::2014-09-25 17:20:08,756::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70095::DEBUG::2014-09-25 17:20:08,757::task::1191::Storage.TaskManager.Task::(prepare) Task=`f6cd18ff-cee7-4bd1-904a-7ba3da39eb5d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70095::DEBUG::2014-09-25 17:20:08,757::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6cd18ff-cee7-4bd1-904a-7ba3da39eb5d`::moving from state preparing -> state finished >Thread-70095::DEBUG::2014-09-25 17:20:08,757::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70095::DEBUG::2014-09-25 17:20:08,757::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70095::DEBUG::2014-09-25 17:20:08,757::task::993::Storage.TaskManager.Task::(_decref) Task=`f6cd18ff-cee7-4bd1-904a-7ba3da39eb5d`::ref 0 aborting False >Thread-70095::DEBUG::2014-09-25 17:20:08,758::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70095::DEBUG::2014-09-25 17:20:08,758::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:08,774::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:08,775::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70096::DEBUG::2014-09-25 17:20:08,775::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70096::DEBUG::2014-09-25 17:20:08,777::task::595::Storage.TaskManager.Task::(_updateState) Task=`7f046ee3-22e5-4ce2-ac23-64c108968b2a`::moving from state init -> state preparing >Thread-70096::INFO::2014-09-25 17:20:08,777::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70096::DEBUG::2014-09-25 17:20:08,778::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`00ce4986-2745-4c09-9f03-579f370542cf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70096::DEBUG::2014-09-25 17:20:08,778::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70096::DEBUG::2014-09-25 17:20:08,779::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70096::DEBUG::2014-09-25 17:20:08,779::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`00ce4986-2745-4c09-9f03-579f370542cf`::Granted request >Thread-70096::DEBUG::2014-09-25 17:20:08,780::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7f046ee3-22e5-4ce2-ac23-64c108968b2a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70096::DEBUG::2014-09-25 17:20:08,780::task::993::Storage.TaskManager.Task::(_decref) Task=`7f046ee3-22e5-4ce2-ac23-64c108968b2a`::ref 1 aborting False >Thread-70096::INFO::2014-09-25 17:20:08,785::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70096::DEBUG::2014-09-25 17:20:08,785::task::1191::Storage.TaskManager.Task::(prepare) Task=`7f046ee3-22e5-4ce2-ac23-64c108968b2a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70096::DEBUG::2014-09-25 17:20:08,785::task::595::Storage.TaskManager.Task::(_updateState) Task=`7f046ee3-22e5-4ce2-ac23-64c108968b2a`::moving from state preparing -> state finished >Thread-70096::DEBUG::2014-09-25 17:20:08,786::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70096::DEBUG::2014-09-25 17:20:08,786::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70096::DEBUG::2014-09-25 17:20:08,786::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70096::DEBUG::2014-09-25 17:20:08,787::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70096::DEBUG::2014-09-25 17:20:08,787::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70096::DEBUG::2014-09-25 17:20:08,787::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70096::DEBUG::2014-09-25 17:20:08,787::task::993::Storage.TaskManager.Task::(_decref) Task=`7f046ee3-22e5-4ce2-ac23-64c108968b2a`::ref 0 aborting False >Thread-70096::DEBUG::2014-09-25 17:20:08,788::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70096::DEBUG::2014-09-25 17:20:08,788::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:09,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:09,168::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0233579 s, 43.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:09,905::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:09,909::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70097::DEBUG::2014-09-25 17:20:09,914::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:11,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:11,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157929 s, 64.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:12,847::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:12,848::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:12,848::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:12,848::__init__::232::IOProcess::(_processLogs) (107834) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:12,858::__init__::232::IOProcess::(_processLogs) (107834) Queuing response >Thread-79::DEBUG::2014-09-25 17:20:12,859::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:20:12,884::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00822978 s, 68.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:12,885::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:12,885::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:12,885::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:12,886::__init__::232::IOProcess::(_processLogs) (107835) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:12,887::__init__::232::IOProcess::(_processLogs) (107835) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:12,888::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:12,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:12,889::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:12,889::__init__::232::IOProcess::(_processLogs) (107836) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:12,889::__init__::232::IOProcess::(_processLogs) (107836) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:12,890::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:12,891::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:12,891::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:12,891::__init__::232::IOProcess::(_processLogs) (107837) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:12,892::__init__::232::IOProcess::(_processLogs) (107837) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:12,892::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:12,893::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:12,894::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:12,894::__init__::232::IOProcess::(_processLogs) (107838) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:12,894::__init__::232::IOProcess::(_processLogs) (107838) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:12,934::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:12,935::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70098::DEBUG::2014-09-25 17:20:12,938::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:20:13,223::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:13,224::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:13,224::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:13,224::__init__::232::IOProcess::(_processLogs) (107839) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:13,225::__init__::232::IOProcess::(_processLogs) (107839) Queuing response >Thread-80::DEBUG::2014-09-25 17:20:13,226::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:13,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-80::DEBUG::2014-09-25 17:20:13,242::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000470397 s, 804 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:13,243::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:13,243::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:13,243::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:13,244::__init__::232::IOProcess::(_processLogs) (107840) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:13,244::__init__::232::IOProcess::(_processLogs) (107840) Queuing response >Dummy-148::DEBUG::2014-09-25 17:20:13,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0216002 s, 47.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:20:15,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:15,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200883 s, 51.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:15,957::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:15,959::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70099::DEBUG::2014-09-25 17:20:15,961::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:17,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:17,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146479 s, 69.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:18,212::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:18,212::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:18,212::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:18,212::__init__::232::IOProcess::(_processLogs) (107841) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:18,221::__init__::232::IOProcess::(_processLogs) (107841) Queuing response >Thread-81::DEBUG::2014-09-25 17:20:18,222::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:20:18,245::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00580218 s, 63.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:18,246::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:18,247::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:18,247::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:18,247::__init__::232::IOProcess::(_processLogs) (107842) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:18,254::__init__::232::IOProcess::(_processLogs) (107842) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:18,819::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70100::DEBUG::2014-09-25 17:20:18,820::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:20:18,820::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70100::DEBUG::2014-09-25 17:20:18,822::task::595::Storage.TaskManager.Task::(_updateState) Task=`be1d19e8-264f-4748-9f79-e6a61b8135a5`::moving from state init -> state preparing >Thread-70100::INFO::2014-09-25 17:20:18,823::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70100::INFO::2014-09-25 17:20:18,823::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70100::DEBUG::2014-09-25 17:20:18,823::task::1191::Storage.TaskManager.Task::(prepare) Task=`be1d19e8-264f-4748-9f79-e6a61b8135a5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70100::DEBUG::2014-09-25 17:20:18,824::task::595::Storage.TaskManager.Task::(_updateState) Task=`be1d19e8-264f-4748-9f79-e6a61b8135a5`::moving from state preparing -> state finished >Thread-70100::DEBUG::2014-09-25 17:20:18,824::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70100::DEBUG::2014-09-25 17:20:18,824::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70100::DEBUG::2014-09-25 17:20:18,824::task::993::Storage.TaskManager.Task::(_decref) Task=`be1d19e8-264f-4748-9f79-e6a61b8135a5`::ref 0 aborting False >Thread-70100::DEBUG::2014-09-25 17:20:18,825::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70100::DEBUG::2014-09-25 17:20:18,825::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:18,842::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:18,843::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70101::DEBUG::2014-09-25 17:20:18,843::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70101::DEBUG::2014-09-25 17:20:18,844::task::595::Storage.TaskManager.Task::(_updateState) Task=`f97b1c33-b15f-4453-9663-03a763235403`::moving from state init -> state preparing >Thread-70101::INFO::2014-09-25 17:20:18,845::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70101::DEBUG::2014-09-25 17:20:18,845::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9a122101-ce37-41c8-b61c-fd757a2dc2e7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70101::DEBUG::2014-09-25 17:20:18,846::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70101::DEBUG::2014-09-25 17:20:18,846::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70101::DEBUG::2014-09-25 17:20:18,846::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9a122101-ce37-41c8-b61c-fd757a2dc2e7`::Granted request >Thread-70101::DEBUG::2014-09-25 17:20:18,848::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f97b1c33-b15f-4453-9663-03a763235403`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70101::DEBUG::2014-09-25 17:20:18,848::task::993::Storage.TaskManager.Task::(_decref) Task=`f97b1c33-b15f-4453-9663-03a763235403`::ref 1 aborting False >Thread-70101::INFO::2014-09-25 17:20:18,853::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70101::DEBUG::2014-09-25 17:20:18,853::task::1191::Storage.TaskManager.Task::(prepare) Task=`f97b1c33-b15f-4453-9663-03a763235403`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70101::DEBUG::2014-09-25 17:20:18,853::task::595::Storage.TaskManager.Task::(_updateState) Task=`f97b1c33-b15f-4453-9663-03a763235403`::moving from state preparing -> state finished >Thread-70101::DEBUG::2014-09-25 17:20:18,853::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70101::DEBUG::2014-09-25 17:20:18,854::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70101::DEBUG::2014-09-25 17:20:18,854::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70101::DEBUG::2014-09-25 17:20:18,854::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70101::DEBUG::2014-09-25 17:20:18,854::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70101::DEBUG::2014-09-25 17:20:18,855::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70101::DEBUG::2014-09-25 17:20:18,855::task::993::Storage.TaskManager.Task::(_decref) Task=`f97b1c33-b15f-4453-9663-03a763235403`::ref 0 aborting False >Thread-70101::DEBUG::2014-09-25 17:20:18,855::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70101::DEBUG::2014-09-25 17:20:18,856::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:18,978::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:18,979::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70102::DEBUG::2014-09-25 17:20:18,981::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:19,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:19,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180761 s, 56.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:20:21,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:21,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130309 s, 78.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:21,996::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:21,997::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70103::DEBUG::2014-09-25 17:20:22,000::task::595::Storage.TaskManager.Task::(_updateState) Task=`7c8fc5b9-9866-44dc-905c-0e12dafe1745`::moving from state init -> state preparing >Thread-70103::INFO::2014-09-25 17:20:22,000::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70103::INFO::2014-09-25 17:20:22,001::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00822978', 'lastCheck': '9.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000470397', 'lastCheck': '8.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00580218', 'lastCheck': '3.7', 'valid': True}} >Thread-70103::DEBUG::2014-09-25 17:20:22,001::task::1191::Storage.TaskManager.Task::(prepare) Task=`7c8fc5b9-9866-44dc-905c-0e12dafe1745`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00822978', 'lastCheck': '9.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000470397', 'lastCheck': '8.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00580218', 'lastCheck': '3.7', 'valid': True}} >Thread-70103::DEBUG::2014-09-25 17:20:22,001::task::595::Storage.TaskManager.Task::(_updateState) Task=`7c8fc5b9-9866-44dc-905c-0e12dafe1745`::moving from state preparing -> state finished >Thread-70103::DEBUG::2014-09-25 17:20:22,001::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70103::DEBUG::2014-09-25 17:20:22,001::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70103::DEBUG::2014-09-25 17:20:22,002::task::993::Storage.TaskManager.Task::(_decref) Task=`7c8fc5b9-9866-44dc-905c-0e12dafe1745`::ref 0 aborting False >Thread-70103::DEBUG::2014-09-25 17:20:22,004::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:22,029::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:22,030::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70104::DEBUG::2014-09-25 17:20:22,032::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:20:22,894::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:22,895::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:22,895::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:22,895::__init__::232::IOProcess::(_processLogs) (107843) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:22,896::__init__::232::IOProcess::(_processLogs) (107843) Queuing response >Thread-79::DEBUG::2014-09-25 17:20:22,897::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:20:22,910::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00310343 s, 181 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:22,911::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:22,912::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:22,912::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:22,912::__init__::232::IOProcess::(_processLogs) (107844) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:22,913::__init__::232::IOProcess::(_processLogs) (107844) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:22,915::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:22,915::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:22,915::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:22,916::__init__::232::IOProcess::(_processLogs) (107845) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:22,916::__init__::232::IOProcess::(_processLogs) (107845) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:22,917::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:22,917::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:22,918::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:22,918::__init__::232::IOProcess::(_processLogs) (107846) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:22,918::__init__::232::IOProcess::(_processLogs) (107846) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:22,919::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:22,919::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:22,920::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:22,920::__init__::232::IOProcess::(_processLogs) (107847) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:22,921::__init__::232::IOProcess::(_processLogs) (107847) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:23,246::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:23,247::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:23,247::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:23,247::__init__::232::IOProcess::(_processLogs) (107848) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:23,248::__init__::232::IOProcess::(_processLogs) (107848) Queuing response >Thread-80::DEBUG::2014-09-25 17:20:23,249::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:20:23,257::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000376807 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:23,258::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:23,259::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:23,259::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:23,259::__init__::232::IOProcess::(_processLogs) (107849) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:23,259::__init__::232::IOProcess::(_processLogs) (107849) Queuing response >Dummy-148::DEBUG::2014-09-25 17:20:23,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:23,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180099 s, 56.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:25,100::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:25,101::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70105::DEBUG::2014-09-25 17:20:25,103::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:25,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:25,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.023294 s, 44.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:20:27,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:27,648::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.01856 s, 55.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:28,118::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:28,119::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70106::DEBUG::2014-09-25 17:20:28,121::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:20:28,256::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:28,257::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:28,257::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:28,257::__init__::232::IOProcess::(_processLogs) (107850) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:28,260::__init__::232::IOProcess::(_processLogs) (107850) Queuing response >Thread-81::DEBUG::2014-09-25 17:20:28,261::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:20:28,274::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00146414 s, 253 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:28,275::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:28,276::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:28,276::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:28,276::__init__::232::IOProcess::(_processLogs) (107851) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:28,278::__init__::232::IOProcess::(_processLogs) (107851) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:28,885::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:28,887::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70107::DEBUG::2014-09-25 17:20:28,887::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70107::DEBUG::2014-09-25 17:20:28,889::task::595::Storage.TaskManager.Task::(_updateState) Task=`f28d9492-93b7-4b49-b59f-14c2b7d9b82f`::moving from state init -> state preparing >Thread-70107::INFO::2014-09-25 17:20:28,889::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70107::INFO::2014-09-25 17:20:28,890::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70107::DEBUG::2014-09-25 17:20:28,890::task::1191::Storage.TaskManager.Task::(prepare) Task=`f28d9492-93b7-4b49-b59f-14c2b7d9b82f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70107::DEBUG::2014-09-25 17:20:28,890::task::595::Storage.TaskManager.Task::(_updateState) Task=`f28d9492-93b7-4b49-b59f-14c2b7d9b82f`::moving from state preparing -> state finished >Thread-70107::DEBUG::2014-09-25 17:20:28,890::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70107::DEBUG::2014-09-25 17:20:28,891::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70107::DEBUG::2014-09-25 17:20:28,891::task::993::Storage.TaskManager.Task::(_decref) Task=`f28d9492-93b7-4b49-b59f-14c2b7d9b82f`::ref 0 aborting False >Thread-70107::DEBUG::2014-09-25 17:20:28,891::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70107::DEBUG::2014-09-25 17:20:28,892::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:28,905::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:28,906::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70108::DEBUG::2014-09-25 17:20:28,906::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70108::DEBUG::2014-09-25 17:20:28,908::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b336c95-5457-4e46-a17f-342643cfea09`::moving from state init -> state preparing >Thread-70108::INFO::2014-09-25 17:20:28,908::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70108::DEBUG::2014-09-25 17:20:28,909::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9e37346d-afc1-45b9-9851-0ff6858432c0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70108::DEBUG::2014-09-25 17:20:28,909::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70108::DEBUG::2014-09-25 17:20:28,909::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70108::DEBUG::2014-09-25 17:20:28,910::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9e37346d-afc1-45b9-9851-0ff6858432c0`::Granted request >Thread-70108::DEBUG::2014-09-25 17:20:28,910::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4b336c95-5457-4e46-a17f-342643cfea09`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70108::DEBUG::2014-09-25 17:20:28,910::task::993::Storage.TaskManager.Task::(_decref) Task=`4b336c95-5457-4e46-a17f-342643cfea09`::ref 1 aborting False >Thread-70108::INFO::2014-09-25 17:20:28,915::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70108::DEBUG::2014-09-25 17:20:28,915::task::1191::Storage.TaskManager.Task::(prepare) Task=`4b336c95-5457-4e46-a17f-342643cfea09`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70108::DEBUG::2014-09-25 17:20:28,915::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b336c95-5457-4e46-a17f-342643cfea09`::moving from state preparing -> state finished >Thread-70108::DEBUG::2014-09-25 17:20:28,915::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70108::DEBUG::2014-09-25 17:20:28,916::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70108::DEBUG::2014-09-25 17:20:28,916::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70108::DEBUG::2014-09-25 17:20:28,916::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70108::DEBUG::2014-09-25 17:20:28,917::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70108::DEBUG::2014-09-25 17:20:28,917::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70108::DEBUG::2014-09-25 17:20:28,917::task::993::Storage.TaskManager.Task::(_decref) Task=`4b336c95-5457-4e46-a17f-342643cfea09`::ref 0 aborting False >Thread-70108::DEBUG::2014-09-25 17:20:28,918::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70108::DEBUG::2014-09-25 17:20:28,918::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:29,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:29,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157308 s, 65.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:31,137::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:31,138::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70109::DEBUG::2014-09-25 17:20:31,140::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:31,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:31,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192297 s, 53.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:32,921::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:32,921::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:32,922::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:32,922::__init__::232::IOProcess::(_processLogs) (107852) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:32,934::__init__::232::IOProcess::(_processLogs) (107852) Queuing response >Thread-79::DEBUG::2014-09-25 17:20:32,936::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:20:32,960::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00758997 s, 74.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:32,961::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:32,962::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:32,962::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:32,962::__init__::232::IOProcess::(_processLogs) (107853) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:32,970::__init__::232::IOProcess::(_processLogs) (107853) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:32,971::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:32,972::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:32,972::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:32,973::__init__::232::IOProcess::(_processLogs) (107854) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:32,973::__init__::232::IOProcess::(_processLogs) (107854) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:32,974::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:32,974::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:32,974::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:32,975::__init__::232::IOProcess::(_processLogs) (107855) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:32,975::__init__::232::IOProcess::(_processLogs) (107855) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:32,976::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:32,976::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:32,977::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:32,977::__init__::232::IOProcess::(_processLogs) (107856) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:32,978::__init__::232::IOProcess::(_processLogs) (107856) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:33,262::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:33,262::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:33,262::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:33,262::__init__::232::IOProcess::(_processLogs) (107857) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:33,263::__init__::232::IOProcess::(_processLogs) (107857) Queuing response >Thread-80::DEBUG::2014-09-25 17:20:33,264::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:20:33,273::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000595014 s, 635 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:33,274::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:33,274::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:33,274::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:33,274::__init__::232::IOProcess::(_processLogs) (107858) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:33,275::__init__::232::IOProcess::(_processLogs) (107858) Queuing response >Dummy-148::DEBUG::2014-09-25 17:20:33,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:33,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013178 s, 77.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:34,154::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:34,155::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70110::DEBUG::2014-09-25 17:20:34,158::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:35,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:35,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199067 s, 51.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:37,172::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:37,173::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70111::DEBUG::2014-09-25 17:20:37,176::task::595::Storage.TaskManager.Task::(_updateState) Task=`c98b6872-bd4a-47aa-86b9-bb86f04f106b`::moving from state init -> state preparing >Thread-70111::INFO::2014-09-25 17:20:37,176::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70111::INFO::2014-09-25 17:20:37,177::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00758997', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000595014', 'lastCheck': '3.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00146414', 'lastCheck': '8.9', 'valid': True}} >Thread-70111::DEBUG::2014-09-25 17:20:37,177::task::1191::Storage.TaskManager.Task::(prepare) Task=`c98b6872-bd4a-47aa-86b9-bb86f04f106b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00758997', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000595014', 'lastCheck': '3.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00146414', 'lastCheck': '8.9', 'valid': True}} >Thread-70111::DEBUG::2014-09-25 17:20:37,177::task::595::Storage.TaskManager.Task::(_updateState) Task=`c98b6872-bd4a-47aa-86b9-bb86f04f106b`::moving from state preparing -> state finished >Thread-70111::DEBUG::2014-09-25 17:20:37,178::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70111::DEBUG::2014-09-25 17:20:37,178::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70111::DEBUG::2014-09-25 17:20:37,178::task::993::Storage.TaskManager.Task::(_decref) Task=`c98b6872-bd4a-47aa-86b9-bb86f04f106b`::ref 0 aborting False >Thread-70111::DEBUG::2014-09-25 17:20:37,180::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:37,201::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:37,202::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70112::DEBUG::2014-09-25 17:20:37,204::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:37,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:37,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160936 s, 63.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:38,280::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:38,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:38,280::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:38,281::__init__::232::IOProcess::(_processLogs) (107859) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:38,285::__init__::232::IOProcess::(_processLogs) (107859) Queuing response >Thread-81::DEBUG::2014-09-25 17:20:38,287::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:20:38,299::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00217615 s, 170 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:38,300::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:38,300::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:38,301::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:38,301::__init__::232::IOProcess::(_processLogs) (107860) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:38,304::__init__::232::IOProcess::(_processLogs) (107860) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:38,947::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70113::DEBUG::2014-09-25 17:20:38,949::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:20:38,949::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70113::DEBUG::2014-09-25 17:20:38,951::task::595::Storage.TaskManager.Task::(_updateState) Task=`e086cf12-409e-4030-9450-00e3b1790bfa`::moving from state init -> state preparing >Thread-70113::INFO::2014-09-25 17:20:38,951::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70113::INFO::2014-09-25 17:20:38,952::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70113::DEBUG::2014-09-25 17:20:38,952::task::1191::Storage.TaskManager.Task::(prepare) Task=`e086cf12-409e-4030-9450-00e3b1790bfa`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70113::DEBUG::2014-09-25 17:20:38,952::task::595::Storage.TaskManager.Task::(_updateState) Task=`e086cf12-409e-4030-9450-00e3b1790bfa`::moving from state preparing -> state finished >Thread-70113::DEBUG::2014-09-25 17:20:38,953::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70113::DEBUG::2014-09-25 17:20:38,953::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70113::DEBUG::2014-09-25 17:20:38,953::task::993::Storage.TaskManager.Task::(_decref) Task=`e086cf12-409e-4030-9450-00e3b1790bfa`::ref 0 aborting False >Thread-70113::DEBUG::2014-09-25 17:20:38,953::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70113::DEBUG::2014-09-25 17:20:38,954::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:38,967::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:38,969::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70114::DEBUG::2014-09-25 17:20:38,969::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70114::DEBUG::2014-09-25 17:20:38,971::task::595::Storage.TaskManager.Task::(_updateState) Task=`5021af50-6744-4c56-8f2b-970b2c5bcf53`::moving from state init -> state preparing >Thread-70114::INFO::2014-09-25 17:20:38,971::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70114::DEBUG::2014-09-25 17:20:38,972::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4f30c643-5703-4494-965b-0df97870df8f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70114::DEBUG::2014-09-25 17:20:38,972::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70114::DEBUG::2014-09-25 17:20:38,972::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70114::DEBUG::2014-09-25 17:20:38,972::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4f30c643-5703-4494-965b-0df97870df8f`::Granted request >Thread-70114::DEBUG::2014-09-25 17:20:38,973::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5021af50-6744-4c56-8f2b-970b2c5bcf53`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70114::DEBUG::2014-09-25 17:20:38,973::task::993::Storage.TaskManager.Task::(_decref) Task=`5021af50-6744-4c56-8f2b-970b2c5bcf53`::ref 1 aborting False >Thread-70114::INFO::2014-09-25 17:20:38,978::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70114::DEBUG::2014-09-25 17:20:38,978::task::1191::Storage.TaskManager.Task::(prepare) Task=`5021af50-6744-4c56-8f2b-970b2c5bcf53`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70114::DEBUG::2014-09-25 17:20:38,978::task::595::Storage.TaskManager.Task::(_updateState) Task=`5021af50-6744-4c56-8f2b-970b2c5bcf53`::moving from state preparing -> state finished >Thread-70114::DEBUG::2014-09-25 17:20:38,978::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70114::DEBUG::2014-09-25 17:20:38,979::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70114::DEBUG::2014-09-25 17:20:38,979::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70114::DEBUG::2014-09-25 17:20:38,979::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70114::DEBUG::2014-09-25 17:20:38,980::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70114::DEBUG::2014-09-25 17:20:38,980::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70114::DEBUG::2014-09-25 17:20:38,980::task::993::Storage.TaskManager.Task::(_decref) Task=`5021af50-6744-4c56-8f2b-970b2c5bcf53`::ref 0 aborting False >Thread-70114::DEBUG::2014-09-25 17:20:38,981::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70114::DEBUG::2014-09-25 17:20:38,981::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:39,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:39,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131542 s, 77.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:40,273::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:40,275::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70115::DEBUG::2014-09-25 17:20:40,277::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:41,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:41,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166907 s, 61.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:42,978::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:42,979::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:42,979::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:42,979::__init__::232::IOProcess::(_processLogs) (107861) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:42,985::__init__::232::IOProcess::(_processLogs) (107861) Queuing response >Thread-79::DEBUG::2014-09-25 17:20:42,987::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:20:43,009::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00738638 s, 76.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:43,010::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:43,011::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:43,011::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:43,011::__init__::232::IOProcess::(_processLogs) (107862) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:43,018::__init__::232::IOProcess::(_processLogs) (107862) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:43,019::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:43,020::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:43,020::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:43,020::__init__::232::IOProcess::(_processLogs) (107863) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:43,026::__init__::232::IOProcess::(_processLogs) (107863) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:43,027::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:43,027::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:43,027::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:43,028::__init__::232::IOProcess::(_processLogs) (107864) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:43,028::__init__::232::IOProcess::(_processLogs) (107864) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:43,029::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:43,029::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:43,029::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:43,030::__init__::232::IOProcess::(_processLogs) (107865) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:43,030::__init__::232::IOProcess::(_processLogs) (107865) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:43,277::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:43,277::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:43,278::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:43,278::__init__::232::IOProcess::(_processLogs) (107866) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:43,279::__init__::232::IOProcess::(_processLogs) (107866) Queuing response >Thread-80::DEBUG::2014-09-25 17:20:43,280::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:20:43,289::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000433955 s, 871 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:43,290::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:43,290::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:43,290::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:43,291::__init__::232::IOProcess::(_processLogs) (107867) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:43,291::__init__::232::IOProcess::(_processLogs) (107867) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:43,297::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:43,298::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70116::DEBUG::2014-09-25 17:20:43,301::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:44,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:44,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130676 s, 78.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:20:46,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:46,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0304656 s, 33.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:46,318::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:46,322::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70117::DEBUG::2014-09-25 17:20:46,331::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:48,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:20:48,311::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:48,311::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:48,311::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:48,312::__init__::232::IOProcess::(_processLogs) (107868) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:48,325::__init__::232::IOProcess::(_processLogs) (107868) Queuing response >Thread-81::DEBUG::2014-09-25 17:20:48,327::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:48,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0273814 s, 37.4 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 17:20:48,342::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.002408 s, 154 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:48,344::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:48,345::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:48,346::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:48,347::__init__::232::IOProcess::(_processLogs) (107869) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:48,348::__init__::232::IOProcess::(_processLogs) (107869) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:49,011::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70118::DEBUG::2014-09-25 17:20:49,013::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:20:49,013::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70118::DEBUG::2014-09-25 17:20:49,015::task::595::Storage.TaskManager.Task::(_updateState) Task=`48ffe74f-37de-4b60-a846-5c6055ddf2b2`::moving from state init -> state preparing >Thread-70118::INFO::2014-09-25 17:20:49,015::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70118::INFO::2014-09-25 17:20:49,016::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70118::DEBUG::2014-09-25 17:20:49,016::task::1191::Storage.TaskManager.Task::(prepare) Task=`48ffe74f-37de-4b60-a846-5c6055ddf2b2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70118::DEBUG::2014-09-25 17:20:49,016::task::595::Storage.TaskManager.Task::(_updateState) Task=`48ffe74f-37de-4b60-a846-5c6055ddf2b2`::moving from state preparing -> state finished >Thread-70118::DEBUG::2014-09-25 17:20:49,017::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70118::DEBUG::2014-09-25 17:20:49,017::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70118::DEBUG::2014-09-25 17:20:49,017::task::993::Storage.TaskManager.Task::(_decref) Task=`48ffe74f-37de-4b60-a846-5c6055ddf2b2`::ref 0 aborting False >Thread-70118::DEBUG::2014-09-25 17:20:49,017::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70118::DEBUG::2014-09-25 17:20:49,018::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:49,031::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70119::DEBUG::2014-09-25 17:20:49,032::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:20:49,033::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70119::DEBUG::2014-09-25 17:20:49,035::task::595::Storage.TaskManager.Task::(_updateState) Task=`f7590328-dcdd-4f62-8c88-445e0e9f793d`::moving from state init -> state preparing >Thread-70119::INFO::2014-09-25 17:20:49,035::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70119::DEBUG::2014-09-25 17:20:49,036::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`86b008be-4f84-4882-99f1-d368d6accec6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70119::DEBUG::2014-09-25 17:20:49,036::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70119::DEBUG::2014-09-25 17:20:49,036::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70119::DEBUG::2014-09-25 17:20:49,036::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`86b008be-4f84-4882-99f1-d368d6accec6`::Granted request >Thread-70119::DEBUG::2014-09-25 17:20:49,037::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f7590328-dcdd-4f62-8c88-445e0e9f793d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70119::DEBUG::2014-09-25 17:20:49,037::task::993::Storage.TaskManager.Task::(_decref) Task=`f7590328-dcdd-4f62-8c88-445e0e9f793d`::ref 1 aborting False >Thread-70119::INFO::2014-09-25 17:20:49,042::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70119::DEBUG::2014-09-25 17:20:49,042::task::1191::Storage.TaskManager.Task::(prepare) Task=`f7590328-dcdd-4f62-8c88-445e0e9f793d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70119::DEBUG::2014-09-25 17:20:49,042::task::595::Storage.TaskManager.Task::(_updateState) Task=`f7590328-dcdd-4f62-8c88-445e0e9f793d`::moving from state preparing -> state finished >Thread-70119::DEBUG::2014-09-25 17:20:49,042::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70119::DEBUG::2014-09-25 17:20:49,043::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70119::DEBUG::2014-09-25 17:20:49,043::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70119::DEBUG::2014-09-25 17:20:49,043::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70119::DEBUG::2014-09-25 17:20:49,043::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70119::DEBUG::2014-09-25 17:20:49,044::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70119::DEBUG::2014-09-25 17:20:49,044::task::993::Storage.TaskManager.Task::(_decref) Task=`f7590328-dcdd-4f62-8c88-445e0e9f793d`::ref 0 aborting False >Thread-70119::DEBUG::2014-09-25 17:20:49,044::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70119::DEBUG::2014-09-25 17:20:49,045::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:49,351::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:49,352::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70120::DEBUG::2014-09-25 17:20:49,354::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:50,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:50,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171477 s, 59.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:52,370::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:52,372::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70121::DEBUG::2014-09-25 17:20:52,379::task::595::Storage.TaskManager.Task::(_updateState) Task=`d8c378a3-f131-49cb-bd6f-e11f4175db8f`::moving from state init -> state preparing >Thread-70121::INFO::2014-09-25 17:20:52,380::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70121::INFO::2014-09-25 17:20:52,383::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00738638', 'lastCheck': '9.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000433955', 'lastCheck': '9.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.002408', 'lastCheck': '4.0', 'valid': True}} >Thread-70121::DEBUG::2014-09-25 17:20:52,384::task::1191::Storage.TaskManager.Task::(prepare) Task=`d8c378a3-f131-49cb-bd6f-e11f4175db8f`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00738638', 'lastCheck': '9.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000433955', 'lastCheck': '9.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.002408', 'lastCheck': '4.0', 'valid': True}} >Thread-70121::DEBUG::2014-09-25 17:20:52,385::task::595::Storage.TaskManager.Task::(_updateState) Task=`d8c378a3-f131-49cb-bd6f-e11f4175db8f`::moving from state preparing -> state finished >Thread-70121::DEBUG::2014-09-25 17:20:52,386::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70121::DEBUG::2014-09-25 17:20:52,387::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70121::DEBUG::2014-09-25 17:20:52,389::task::993::Storage.TaskManager.Task::(_decref) Task=`d8c378a3-f131-49cb-bd6f-e11f4175db8f`::ref 0 aborting False >Thread-70121::DEBUG::2014-09-25 17:20:52,398::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:52,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:52,426::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:52,430::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70122::DEBUG::2014-09-25 17:20:52,436::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:52,459::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0294141 s, 34.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:53,031::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:53,031::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:53,031::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:53,032::__init__::232::IOProcess::(_processLogs) (107870) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:53,039::__init__::232::IOProcess::(_processLogs) (107870) Queuing response >Thread-79::DEBUG::2014-09-25 17:20:53,040::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:20:53,062::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00180418 s, 311 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:53,063::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:53,063::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:53,064::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:53,064::__init__::232::IOProcess::(_processLogs) (107871) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:53,065::__init__::232::IOProcess::(_processLogs) (107871) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:53,066::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:53,066::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:53,067::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:53,067::__init__::232::IOProcess::(_processLogs) (107872) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:53,068::__init__::232::IOProcess::(_processLogs) (107872) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:53,068::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:53,068::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:53,069::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:53,069::__init__::232::IOProcess::(_processLogs) (107873) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:53,069::__init__::232::IOProcess::(_processLogs) (107873) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:53,070::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:53,070::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:53,071::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:53,072::__init__::232::IOProcess::(_processLogs) (107874) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:20:53,072::__init__::232::IOProcess::(_processLogs) (107874) Queuing response >Thread-29::DEBUG::2014-09-25 17:20:53,294::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:53,294::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:53,294::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:53,294::__init__::232::IOProcess::(_processLogs) (107875) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:53,295::__init__::232::IOProcess::(_processLogs) (107875) Queuing response >Thread-80::DEBUG::2014-09-25 17:20:53,296::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:20:53,305::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000390008 s, 969 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:53,306::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:53,307::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:53,307::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:53,307::__init__::232::IOProcess::(_processLogs) (107876) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:53,307::__init__::232::IOProcess::(_processLogs) (107876) Queuing response >Dummy-148::DEBUG::2014-09-25 17:20:54,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:54,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0258717 s, 39.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:55,495::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:55,497::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70123::DEBUG::2014-09-25 17:20:55,499::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:56,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:56,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133177 s, 76.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:58,351::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:58,351::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:58,352::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:58,352::__init__::232::IOProcess::(_processLogs) (107877) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:58,360::__init__::232::IOProcess::(_processLogs) (107877) Queuing response >Thread-81::DEBUG::2014-09-25 17:20:58,362::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:20:58,392::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00431295 s, 86.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:20:58,393::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:20:58,394::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:20:58,394::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:20:58,394::__init__::232::IOProcess::(_processLogs) (107878) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:20:58,396::__init__::232::IOProcess::(_processLogs) (107878) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:58,516::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:58,517::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70124::DEBUG::2014-09-25 17:20:58,519::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:20:58,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:20:58,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0276212 s, 37.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:59,077::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:20:59,079::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70125::DEBUG::2014-09-25 17:20:59,079::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70125::DEBUG::2014-09-25 17:20:59,081::task::595::Storage.TaskManager.Task::(_updateState) Task=`f0159c21-feba-40ab-be05-85745b0d3d41`::moving from state init -> state preparing >Thread-70125::INFO::2014-09-25 17:20:59,081::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70125::INFO::2014-09-25 17:20:59,082::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70125::DEBUG::2014-09-25 17:20:59,082::task::1191::Storage.TaskManager.Task::(prepare) Task=`f0159c21-feba-40ab-be05-85745b0d3d41`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70125::DEBUG::2014-09-25 17:20:59,082::task::595::Storage.TaskManager.Task::(_updateState) Task=`f0159c21-feba-40ab-be05-85745b0d3d41`::moving from state preparing -> state finished >Thread-70125::DEBUG::2014-09-25 17:20:59,082::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70125::DEBUG::2014-09-25 17:20:59,083::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70125::DEBUG::2014-09-25 17:20:59,083::task::993::Storage.TaskManager.Task::(_decref) Task=`f0159c21-feba-40ab-be05-85745b0d3d41`::ref 0 aborting False >Thread-70125::DEBUG::2014-09-25 17:20:59,083::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70125::DEBUG::2014-09-25 17:20:59,084::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:20:59,096::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70126::DEBUG::2014-09-25 17:20:59,097::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:20:59,097::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70126::DEBUG::2014-09-25 17:20:59,099::task::595::Storage.TaskManager.Task::(_updateState) Task=`c0aec89d-239c-45b9-8d8b-468f991d445d`::moving from state init -> state preparing >Thread-70126::INFO::2014-09-25 17:20:59,099::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70126::DEBUG::2014-09-25 17:20:59,100::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`96b8e96c-c2c2-439d-a270-0993e2fc2782`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70126::DEBUG::2014-09-25 17:20:59,100::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70126::DEBUG::2014-09-25 17:20:59,101::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70126::DEBUG::2014-09-25 17:20:59,101::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`96b8e96c-c2c2-439d-a270-0993e2fc2782`::Granted request >Thread-70126::DEBUG::2014-09-25 17:20:59,102::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c0aec89d-239c-45b9-8d8b-468f991d445d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70126::DEBUG::2014-09-25 17:20:59,102::task::993::Storage.TaskManager.Task::(_decref) Task=`c0aec89d-239c-45b9-8d8b-468f991d445d`::ref 1 aborting False >Thread-70126::INFO::2014-09-25 17:20:59,109::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70126::DEBUG::2014-09-25 17:20:59,109::task::1191::Storage.TaskManager.Task::(prepare) Task=`c0aec89d-239c-45b9-8d8b-468f991d445d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70126::DEBUG::2014-09-25 17:20:59,110::task::595::Storage.TaskManager.Task::(_updateState) Task=`c0aec89d-239c-45b9-8d8b-468f991d445d`::moving from state preparing -> state finished >Thread-70126::DEBUG::2014-09-25 17:20:59,110::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70126::DEBUG::2014-09-25 17:20:59,110::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70126::DEBUG::2014-09-25 17:20:59,110::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70126::DEBUG::2014-09-25 17:20:59,111::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70126::DEBUG::2014-09-25 17:20:59,111::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70126::DEBUG::2014-09-25 17:20:59,111::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70126::DEBUG::2014-09-25 17:20:59,112::task::993::Storage.TaskManager.Task::(_decref) Task=`c0aec89d-239c-45b9-8d8b-468f991d445d`::ref 0 aborting False >Thread-70126::DEBUG::2014-09-25 17:20:59,112::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70126::DEBUG::2014-09-25 17:20:59,113::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:00,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:00,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212784 s, 48.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:01,534::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:01,536::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70127::DEBUG::2014-09-25 17:21:01,538::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:02,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:02,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0128213 s, 79.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:03,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:03,073::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:03,073::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:03,073::__init__::232::IOProcess::(_processLogs) (107879) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:03,076::__init__::232::IOProcess::(_processLogs) (107879) Queuing response >Thread-79::DEBUG::2014-09-25 17:21:03,077::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:21:03,090::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00133001 s, 423 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:03,091::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:03,091::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:03,091::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:03,092::__init__::232::IOProcess::(_processLogs) (107880) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:03,093::__init__::232::IOProcess::(_processLogs) (107880) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:03,094::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:03,095::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:03,096::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:03,096::__init__::232::IOProcess::(_processLogs) (107881) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:03,096::__init__::232::IOProcess::(_processLogs) (107881) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:03,096::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:03,097::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:03,098::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:03,098::__init__::232::IOProcess::(_processLogs) (107882) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:03,098::__init__::232::IOProcess::(_processLogs) (107882) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:03,098::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:03,099::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:03,100::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:03,100::__init__::232::IOProcess::(_processLogs) (107883) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:03,100::__init__::232::IOProcess::(_processLogs) (107883) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:03,309::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:03,309::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:03,309::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:03,309::__init__::232::IOProcess::(_processLogs) (107884) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:03,310::__init__::232::IOProcess::(_processLogs) (107884) Queuing response >Thread-80::DEBUG::2014-09-25 17:21:03,311::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:21:03,320::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000403386 s, 937 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:03,321::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:03,321::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:03,321::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:03,322::__init__::232::IOProcess::(_processLogs) (107885) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:03,322::__init__::232::IOProcess::(_processLogs) (107885) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:04,551::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:04,553::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70128::DEBUG::2014-09-25 17:21:04,554::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:04,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:04,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130673 s, 78.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:21:06,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:06,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209413 s, 48.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:07,569::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:07,571::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70129::DEBUG::2014-09-25 17:21:07,573::task::595::Storage.TaskManager.Task::(_updateState) Task=`2dee51c7-f936-49a8-b418-441cc6a73e7d`::moving from state init -> state preparing >Thread-70129::INFO::2014-09-25 17:21:07,574::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70129::INFO::2014-09-25 17:21:07,574::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00133001', 'lastCheck': '4.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000403386', 'lastCheck': '4.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00431295', 'lastCheck': '9.2', 'valid': True}} >Thread-70129::DEBUG::2014-09-25 17:21:07,575::task::1191::Storage.TaskManager.Task::(prepare) Task=`2dee51c7-f936-49a8-b418-441cc6a73e7d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00133001', 'lastCheck': '4.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000403386', 'lastCheck': '4.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00431295', 'lastCheck': '9.2', 'valid': True}} >Thread-70129::DEBUG::2014-09-25 17:21:07,575::task::595::Storage.TaskManager.Task::(_updateState) Task=`2dee51c7-f936-49a8-b418-441cc6a73e7d`::moving from state preparing -> state finished >Thread-70129::DEBUG::2014-09-25 17:21:07,575::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70129::DEBUG::2014-09-25 17:21:07,575::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70129::DEBUG::2014-09-25 17:21:07,575::task::993::Storage.TaskManager.Task::(_decref) Task=`2dee51c7-f936-49a8-b418-441cc6a73e7d`::ref 0 aborting False >Thread-70129::DEBUG::2014-09-25 17:21:07,578::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:07,602::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:07,603::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70130::DEBUG::2014-09-25 17:21:07,605::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:21:08,398::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:08,398::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:08,398::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:08,399::__init__::232::IOProcess::(_processLogs) (107886) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:08,406::__init__::232::IOProcess::(_processLogs) (107886) Queuing response >Thread-81::DEBUG::2014-09-25 17:21:08,407::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:21:08,432::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00848134 s, 43.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:08,433::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:08,433::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:08,433::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:08,434::__init__::232::IOProcess::(_processLogs) (107887) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:08,439::__init__::232::IOProcess::(_processLogs) (107887) Queuing response >Dummy-148::DEBUG::2014-09-25 17:21:08,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:08,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162916 s, 62.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:09,151::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:09,153::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70131::DEBUG::2014-09-25 17:21:09,153::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70131::DEBUG::2014-09-25 17:21:09,155::task::595::Storage.TaskManager.Task::(_updateState) Task=`cb822bf0-45df-4a2a-8fba-927eb6c6dcf6`::moving from state init -> state preparing >Thread-70131::INFO::2014-09-25 17:21:09,155::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70131::INFO::2014-09-25 17:21:09,156::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70131::DEBUG::2014-09-25 17:21:09,156::task::1191::Storage.TaskManager.Task::(prepare) Task=`cb822bf0-45df-4a2a-8fba-927eb6c6dcf6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70131::DEBUG::2014-09-25 17:21:09,157::task::595::Storage.TaskManager.Task::(_updateState) Task=`cb822bf0-45df-4a2a-8fba-927eb6c6dcf6`::moving from state preparing -> state finished >Thread-70131::DEBUG::2014-09-25 17:21:09,157::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70131::DEBUG::2014-09-25 17:21:09,157::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70131::DEBUG::2014-09-25 17:21:09,157::task::993::Storage.TaskManager.Task::(_decref) Task=`cb822bf0-45df-4a2a-8fba-927eb6c6dcf6`::ref 0 aborting False >Thread-70131::DEBUG::2014-09-25 17:21:09,158::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70131::DEBUG::2014-09-25 17:21:09,158::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:09,180::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:09,181::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70132::DEBUG::2014-09-25 17:21:09,181::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70132::DEBUG::2014-09-25 17:21:09,183::task::595::Storage.TaskManager.Task::(_updateState) Task=`c5e6a777-7028-4241-9e80-67028158d39d`::moving from state init -> state preparing >Thread-70132::INFO::2014-09-25 17:21:09,183::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70132::DEBUG::2014-09-25 17:21:09,184::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2bb65769-bc13-416e-900d-e0d3bbbbd171`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70132::DEBUG::2014-09-25 17:21:09,184::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70132::DEBUG::2014-09-25 17:21:09,184::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70132::DEBUG::2014-09-25 17:21:09,185::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2bb65769-bc13-416e-900d-e0d3bbbbd171`::Granted request >Thread-70132::DEBUG::2014-09-25 17:21:09,185::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c5e6a777-7028-4241-9e80-67028158d39d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70132::DEBUG::2014-09-25 17:21:09,186::task::993::Storage.TaskManager.Task::(_decref) Task=`c5e6a777-7028-4241-9e80-67028158d39d`::ref 1 aborting False >Thread-70132::INFO::2014-09-25 17:21:09,190::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70132::DEBUG::2014-09-25 17:21:09,190::task::1191::Storage.TaskManager.Task::(prepare) Task=`c5e6a777-7028-4241-9e80-67028158d39d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70132::DEBUG::2014-09-25 17:21:09,191::task::595::Storage.TaskManager.Task::(_updateState) Task=`c5e6a777-7028-4241-9e80-67028158d39d`::moving from state preparing -> state finished >Thread-70132::DEBUG::2014-09-25 17:21:09,191::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70132::DEBUG::2014-09-25 17:21:09,191::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70132::DEBUG::2014-09-25 17:21:09,191::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70132::DEBUG::2014-09-25 17:21:09,192::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70132::DEBUG::2014-09-25 17:21:09,192::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70132::DEBUG::2014-09-25 17:21:09,192::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70132::DEBUG::2014-09-25 17:21:09,193::task::993::Storage.TaskManager.Task::(_decref) Task=`c5e6a777-7028-4241-9e80-67028158d39d`::ref 0 aborting False >Thread-70132::DEBUG::2014-09-25 17:21:09,193::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70132::DEBUG::2014-09-25 17:21:09,194::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:10,664::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:10,666::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70133::DEBUG::2014-09-25 17:21:10,667::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:10,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:11,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0227184 s, 45.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:21:13,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:13,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163463 s, 62.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:13,101::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:13,101::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:13,101::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:13,101::__init__::232::IOProcess::(_processLogs) (107888) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:13,104::__init__::232::IOProcess::(_processLogs) (107888) Queuing response >Thread-79::DEBUG::2014-09-25 17:21:13,104::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:21:13,120::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00612518 s, 91.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:13,122::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:13,122::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:13,122::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:13,123::__init__::232::IOProcess::(_processLogs) (107889) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:13,124::__init__::232::IOProcess::(_processLogs) (107889) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:13,125::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:13,125::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:13,125::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:13,126::__init__::232::IOProcess::(_processLogs) (107890) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:13,126::__init__::232::IOProcess::(_processLogs) (107890) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:13,127::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:13,127::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:13,127::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:13,128::__init__::232::IOProcess::(_processLogs) (107891) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:13,128::__init__::232::IOProcess::(_processLogs) (107891) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:13,129::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:13,129::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:13,130::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:13,130::__init__::232::IOProcess::(_processLogs) (107892) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:13,131::__init__::232::IOProcess::(_processLogs) (107892) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:13,324::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:13,325::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:13,325::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:13,325::__init__::232::IOProcess::(_processLogs) (107893) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:13,325::__init__::232::IOProcess::(_processLogs) (107893) Queuing response >Thread-80::DEBUG::2014-09-25 17:21:13,327::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:21:13,335::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000532051 s, 710 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:13,337::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:13,338::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:13,339::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:13,339::__init__::232::IOProcess::(_processLogs) (107894) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:13,339::__init__::232::IOProcess::(_processLogs) (107894) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:13,683::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:13,684::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70134::DEBUG::2014-09-25 17:21:13,686::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:15,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:15,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134477 s, 76.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:16,702::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:16,705::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70135::DEBUG::2014-09-25 17:21:16,710::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:17,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:17,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143087 s, 71.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:18,441::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:18,441::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:18,442::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:18,442::__init__::232::IOProcess::(_processLogs) (107895) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:18,450::__init__::232::IOProcess::(_processLogs) (107895) Queuing response >Thread-81::DEBUG::2014-09-25 17:21:18,452::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:21:18,477::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00119922 s, 309 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:18,478::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:18,479::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:18,479::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:18,479::__init__::232::IOProcess::(_processLogs) (107896) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:18,484::__init__::232::IOProcess::(_processLogs) (107896) Queuing response >Dummy-148::DEBUG::2014-09-25 17:21:19,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:19,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148019 s, 69.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:19,225::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70136::DEBUG::2014-09-25 17:21:19,226::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:21:19,226::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70136::DEBUG::2014-09-25 17:21:19,228::task::595::Storage.TaskManager.Task::(_updateState) Task=`f4fcdcf1-cfa9-4820-b29a-5b73f70e4eaf`::moving from state init -> state preparing >Thread-70136::INFO::2014-09-25 17:21:19,229::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70136::INFO::2014-09-25 17:21:19,229::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70136::DEBUG::2014-09-25 17:21:19,229::task::1191::Storage.TaskManager.Task::(prepare) Task=`f4fcdcf1-cfa9-4820-b29a-5b73f70e4eaf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70136::DEBUG::2014-09-25 17:21:19,230::task::595::Storage.TaskManager.Task::(_updateState) Task=`f4fcdcf1-cfa9-4820-b29a-5b73f70e4eaf`::moving from state preparing -> state finished >Thread-70136::DEBUG::2014-09-25 17:21:19,230::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70136::DEBUG::2014-09-25 17:21:19,230::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70136::DEBUG::2014-09-25 17:21:19,230::task::993::Storage.TaskManager.Task::(_decref) Task=`f4fcdcf1-cfa9-4820-b29a-5b73f70e4eaf`::ref 0 aborting False >Thread-70136::DEBUG::2014-09-25 17:21:19,231::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70136::DEBUG::2014-09-25 17:21:19,231::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:19,243::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:19,245::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70137::DEBUG::2014-09-25 17:21:19,245::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70137::DEBUG::2014-09-25 17:21:19,246::task::595::Storage.TaskManager.Task::(_updateState) Task=`73b343ea-da98-4729-88e9-07c6372c5cfc`::moving from state init -> state preparing >Thread-70137::INFO::2014-09-25 17:21:19,247::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70137::DEBUG::2014-09-25 17:21:19,247::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e3b7f102-650d-4905-9b51-c574bfe6a04d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70137::DEBUG::2014-09-25 17:21:19,248::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70137::DEBUG::2014-09-25 17:21:19,248::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70137::DEBUG::2014-09-25 17:21:19,248::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e3b7f102-650d-4905-9b51-c574bfe6a04d`::Granted request >Thread-70137::DEBUG::2014-09-25 17:21:19,249::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`73b343ea-da98-4729-88e9-07c6372c5cfc`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70137::DEBUG::2014-09-25 17:21:19,249::task::993::Storage.TaskManager.Task::(_decref) Task=`73b343ea-da98-4729-88e9-07c6372c5cfc`::ref 1 aborting False >Thread-70137::INFO::2014-09-25 17:21:19,253::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70137::DEBUG::2014-09-25 17:21:19,254::task::1191::Storage.TaskManager.Task::(prepare) Task=`73b343ea-da98-4729-88e9-07c6372c5cfc`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70137::DEBUG::2014-09-25 17:21:19,254::task::595::Storage.TaskManager.Task::(_updateState) Task=`73b343ea-da98-4729-88e9-07c6372c5cfc`::moving from state preparing -> state finished >Thread-70137::DEBUG::2014-09-25 17:21:19,254::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70137::DEBUG::2014-09-25 17:21:19,254::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70137::DEBUG::2014-09-25 17:21:19,255::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70137::DEBUG::2014-09-25 17:21:19,255::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70137::DEBUG::2014-09-25 17:21:19,255::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70137::DEBUG::2014-09-25 17:21:19,255::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70137::DEBUG::2014-09-25 17:21:19,256::task::993::Storage.TaskManager.Task::(_decref) Task=`73b343ea-da98-4729-88e9-07c6372c5cfc`::ref 0 aborting False >Thread-70137::DEBUG::2014-09-25 17:21:19,256::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70137::DEBUG::2014-09-25 17:21:19,257::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:19,727::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:19,728::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70138::DEBUG::2014-09-25 17:21:19,730::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:21,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:21,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199051 s, 51.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:22,744::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:22,745::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70139::DEBUG::2014-09-25 17:21:22,748::task::595::Storage.TaskManager.Task::(_updateState) Task=`0517a3cb-b5f1-436f-b355-993e67440fbd`::moving from state init -> state preparing >Thread-70139::INFO::2014-09-25 17:21:22,748::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70139::INFO::2014-09-25 17:21:22,748::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00612518', 'lastCheck': '9.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000532051', 'lastCheck': '9.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00119922', 'lastCheck': '4.3', 'valid': True}} >Thread-70139::DEBUG::2014-09-25 17:21:22,749::task::1191::Storage.TaskManager.Task::(prepare) Task=`0517a3cb-b5f1-436f-b355-993e67440fbd`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00612518', 'lastCheck': '9.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000532051', 'lastCheck': '9.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00119922', 'lastCheck': '4.3', 'valid': True}} >Thread-70139::DEBUG::2014-09-25 17:21:22,749::task::595::Storage.TaskManager.Task::(_updateState) Task=`0517a3cb-b5f1-436f-b355-993e67440fbd`::moving from state preparing -> state finished >Thread-70139::DEBUG::2014-09-25 17:21:22,749::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70139::DEBUG::2014-09-25 17:21:22,749::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70139::DEBUG::2014-09-25 17:21:22,750::task::993::Storage.TaskManager.Task::(_decref) Task=`0517a3cb-b5f1-436f-b355-993e67440fbd`::ref 0 aborting False >Thread-70139::DEBUG::2014-09-25 17:21:22,752::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:22,766::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:22,767::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70140::DEBUG::2014-09-25 17:21:22,769::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:21:23,131::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:23,131::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:23,132::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:23,133::__init__::232::IOProcess::(_processLogs) (107897) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 17:21:23,133::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:21:23,134::__init__::232::IOProcess::(_processLogs) (107897) Queuing response >Thread-79::DEBUG::2014-09-25 17:21:23,144::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000441559 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:23,145::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:23,145::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:23,146::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:23,147::__init__::232::IOProcess::(_processLogs) (107898) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:23,147::__init__::232::IOProcess::(_processLogs) (107898) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:23,147::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:23,148::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:23,148::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:23,149::__init__::232::IOProcess::(_processLogs) (107899) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:23,149::__init__::232::IOProcess::(_processLogs) (107899) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:23,150::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:23,150::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:23,150::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:23,151::__init__::232::IOProcess::(_processLogs) (107900) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:23,151::__init__::232::IOProcess::(_processLogs) (107900) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:23,152::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:23,152::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:23,152::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:23,153::__init__::232::IOProcess::(_processLogs) (107901) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:23,154::__init__::232::IOProcess::(_processLogs) (107901) Queuing response >Dummy-148::DEBUG::2014-09-25 17:21:23,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:23,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157459 s, 65.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:23,340::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:23,340::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:23,340::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:23,340::__init__::232::IOProcess::(_processLogs) (107902) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:23,341::__init__::232::IOProcess::(_processLogs) (107902) Queuing response >Thread-80::DEBUG::2014-09-25 17:21:23,342::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:21:23,351::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000448617 s, 843 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:23,352::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:23,352::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:23,352::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:23,353::__init__::232::IOProcess::(_processLogs) (107903) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:23,353::__init__::232::IOProcess::(_processLogs) (107903) Queuing response >Dummy-148::DEBUG::2014-09-25 17:21:25,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:25,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204712 s, 50.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:25,834::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:25,836::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70141::DEBUG::2014-09-25 17:21:25,838::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:27,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:27,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020333 s, 50.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:28,486::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:28,487::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:28,487::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:28,487::__init__::232::IOProcess::(_processLogs) (107904) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:28,487::__init__::232::IOProcess::(_processLogs) (107904) Queuing response >Thread-81::DEBUG::2014-09-25 17:21:28,489::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:21:28,497::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000409276 s, 906 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:28,498::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:28,499::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:28,499::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:28,499::__init__::232::IOProcess::(_processLogs) (107905) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:28,500::__init__::232::IOProcess::(_processLogs) (107905) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:28,851::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:28,852::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70142::DEBUG::2014-09-25 17:21:28,854::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:29,284::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70143::DEBUG::2014-09-25 17:21:29,285::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:21:29,285::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70143::DEBUG::2014-09-25 17:21:29,287::task::595::Storage.TaskManager.Task::(_updateState) Task=`9c5d08f9-4e47-4387-a9be-9b56f8aafcf2`::moving from state init -> state preparing >Thread-70143::INFO::2014-09-25 17:21:29,288::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70143::INFO::2014-09-25 17:21:29,288::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70143::DEBUG::2014-09-25 17:21:29,288::task::1191::Storage.TaskManager.Task::(prepare) Task=`9c5d08f9-4e47-4387-a9be-9b56f8aafcf2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70143::DEBUG::2014-09-25 17:21:29,289::task::595::Storage.TaskManager.Task::(_updateState) Task=`9c5d08f9-4e47-4387-a9be-9b56f8aafcf2`::moving from state preparing -> state finished >Thread-70143::DEBUG::2014-09-25 17:21:29,289::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70143::DEBUG::2014-09-25 17:21:29,289::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70143::DEBUG::2014-09-25 17:21:29,289::task::993::Storage.TaskManager.Task::(_decref) Task=`9c5d08f9-4e47-4387-a9be-9b56f8aafcf2`::ref 0 aborting False >Thread-70143::DEBUG::2014-09-25 17:21:29,290::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70143::DEBUG::2014-09-25 17:21:29,290::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:29,300::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:29,301::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70144::DEBUG::2014-09-25 17:21:29,301::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70144::DEBUG::2014-09-25 17:21:29,303::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b5b1d02-233b-4b4e-b3ed-86107ed4e640`::moving from state init -> state preparing >Thread-70144::INFO::2014-09-25 17:21:29,303::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70144::DEBUG::2014-09-25 17:21:29,304::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`52ca4d8e-3f1b-4c53-b527-29326f583acd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70144::DEBUG::2014-09-25 17:21:29,304::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70144::DEBUG::2014-09-25 17:21:29,304::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70144::DEBUG::2014-09-25 17:21:29,305::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`52ca4d8e-3f1b-4c53-b527-29326f583acd`::Granted request >Thread-70144::DEBUG::2014-09-25 17:21:29,305::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1b5b1d02-233b-4b4e-b3ed-86107ed4e640`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70144::DEBUG::2014-09-25 17:21:29,305::task::993::Storage.TaskManager.Task::(_decref) Task=`1b5b1d02-233b-4b4e-b3ed-86107ed4e640`::ref 1 aborting False >Thread-70144::INFO::2014-09-25 17:21:29,310::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70144::DEBUG::2014-09-25 17:21:29,310::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b5b1d02-233b-4b4e-b3ed-86107ed4e640`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70144::DEBUG::2014-09-25 17:21:29,310::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b5b1d02-233b-4b4e-b3ed-86107ed4e640`::moving from state preparing -> state finished >Thread-70144::DEBUG::2014-09-25 17:21:29,310::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70144::DEBUG::2014-09-25 17:21:29,311::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70144::DEBUG::2014-09-25 17:21:29,311::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70144::DEBUG::2014-09-25 17:21:29,311::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70144::DEBUG::2014-09-25 17:21:29,311::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70144::DEBUG::2014-09-25 17:21:29,312::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70144::DEBUG::2014-09-25 17:21:29,312::task::993::Storage.TaskManager.Task::(_decref) Task=`1b5b1d02-233b-4b4e-b3ed-86107ed4e640`::ref 0 aborting False >Thread-70144::DEBUG::2014-09-25 17:21:29,312::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70144::DEBUG::2014-09-25 17:21:29,313::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:29,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:29,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193493 s, 52.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:21:31,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:31,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207704 s, 49.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:31,866::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:31,867::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70145::DEBUG::2014-09-25 17:21:31,869::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:21:33,154::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:33,155::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:33,155::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:33,155::__init__::232::IOProcess::(_processLogs) (107906) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:33,160::__init__::232::IOProcess::(_processLogs) (107906) Queuing response >Thread-79::DEBUG::2014-09-25 17:21:33,162::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:21:33,186::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00618765 s, 90.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:33,187::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:33,188::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:33,188::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:33,188::__init__::232::IOProcess::(_processLogs) (107907) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:33,195::__init__::232::IOProcess::(_processLogs) (107907) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:33,196::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:33,196::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:33,197::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:33,197::__init__::232::IOProcess::(_processLogs) (107908) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:33,197::__init__::232::IOProcess::(_processLogs) (107908) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:33,198::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:33,198::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:33,199::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:33,199::__init__::232::IOProcess::(_processLogs) (107909) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:33,200::__init__::232::IOProcess::(_processLogs) (107909) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:33,200::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:33,201::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:33,202::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:33,202::__init__::232::IOProcess::(_processLogs) (107910) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:33,202::__init__::232::IOProcess::(_processLogs) (107910) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:33,355::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:33,356::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:33,356::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:33,356::__init__::232::IOProcess::(_processLogs) (107911) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:33,357::__init__::232::IOProcess::(_processLogs) (107911) Queuing response >Thread-80::DEBUG::2014-09-25 17:21:33,358::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:21:33,367::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000495811 s, 762 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:33,368::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:33,368::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:33,369::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:33,369::__init__::232::IOProcess::(_processLogs) (107912) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:33,369::__init__::232::IOProcess::(_processLogs) (107912) Queuing response >Dummy-148::DEBUG::2014-09-25 17:21:33,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:33,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173203 s, 59.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:34,882::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:34,883::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70146::DEBUG::2014-09-25 17:21:34,885::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:35,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:35,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175911 s, 58.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:21:37,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:37,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146244 s, 70.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:37,898::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:37,899::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70147::DEBUG::2014-09-25 17:21:37,902::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c923cf9-8e34-4d26-b3d5-bab0718a75b3`::moving from state init -> state preparing >Thread-70147::INFO::2014-09-25 17:21:37,902::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70147::INFO::2014-09-25 17:21:37,903::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00618765', 'lastCheck': '4.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000495811', 'lastCheck': '4.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409276', 'lastCheck': '9.4', 'valid': True}} >Thread-70147::DEBUG::2014-09-25 17:21:37,903::task::1191::Storage.TaskManager.Task::(prepare) Task=`6c923cf9-8e34-4d26-b3d5-bab0718a75b3`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00618765', 'lastCheck': '4.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000495811', 'lastCheck': '4.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409276', 'lastCheck': '9.4', 'valid': True}} >Thread-70147::DEBUG::2014-09-25 17:21:37,903::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c923cf9-8e34-4d26-b3d5-bab0718a75b3`::moving from state preparing -> state finished >Thread-70147::DEBUG::2014-09-25 17:21:37,904::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70147::DEBUG::2014-09-25 17:21:37,904::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70147::DEBUG::2014-09-25 17:21:37,904::task::993::Storage.TaskManager.Task::(_decref) Task=`6c923cf9-8e34-4d26-b3d5-bab0718a75b3`::ref 0 aborting False >Thread-70147::DEBUG::2014-09-25 17:21:37,906::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:37,922::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:37,923::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70148::DEBUG::2014-09-25 17:21:37,925::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:21:38,502::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:38,503::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:38,503::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:38,503::__init__::232::IOProcess::(_processLogs) (107913) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:38,622::__init__::232::IOProcess::(_processLogs) (107913) Queuing response >Thread-81::DEBUG::2014-09-25 17:21:38,623::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:21:38,637::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00223914 s, 166 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:38,638::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:38,638::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:38,639::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:38,639::__init__::232::IOProcess::(_processLogs) (107914) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:38,640::__init__::232::IOProcess::(_processLogs) (107914) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:39,344::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70149::DEBUG::2014-09-25 17:21:39,345::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:21:39,345::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70149::DEBUG::2014-09-25 17:21:39,348::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b2c8217-64f2-4bc5-bcb3-0fda5a7a6c09`::moving from state init -> state preparing >Thread-70149::INFO::2014-09-25 17:21:39,348::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70149::INFO::2014-09-25 17:21:39,348::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70149::DEBUG::2014-09-25 17:21:39,349::task::1191::Storage.TaskManager.Task::(prepare) Task=`4b2c8217-64f2-4bc5-bcb3-0fda5a7a6c09`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70149::DEBUG::2014-09-25 17:21:39,349::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b2c8217-64f2-4bc5-bcb3-0fda5a7a6c09`::moving from state preparing -> state finished >Thread-70149::DEBUG::2014-09-25 17:21:39,349::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70149::DEBUG::2014-09-25 17:21:39,349::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70149::DEBUG::2014-09-25 17:21:39,350::task::993::Storage.TaskManager.Task::(_decref) Task=`4b2c8217-64f2-4bc5-bcb3-0fda5a7a6c09`::ref 0 aborting False >Thread-70149::DEBUG::2014-09-25 17:21:39,350::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70149::DEBUG::2014-09-25 17:21:39,350::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:39,364::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70150::DEBUG::2014-09-25 17:21:39,366::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:21:39,366::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70150::DEBUG::2014-09-25 17:21:39,368::task::595::Storage.TaskManager.Task::(_updateState) Task=`4d491209-6d13-429f-8634-fefa562a1d30`::moving from state init -> state preparing >Thread-70150::INFO::2014-09-25 17:21:39,368::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70150::DEBUG::2014-09-25 17:21:39,369::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`581bb082-2269-42df-ad40-f09695614321`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70150::DEBUG::2014-09-25 17:21:39,369::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70150::DEBUG::2014-09-25 17:21:39,369::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70150::DEBUG::2014-09-25 17:21:39,370::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`581bb082-2269-42df-ad40-f09695614321`::Granted request >Thread-70150::DEBUG::2014-09-25 17:21:39,370::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4d491209-6d13-429f-8634-fefa562a1d30`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70150::DEBUG::2014-09-25 17:21:39,370::task::993::Storage.TaskManager.Task::(_decref) Task=`4d491209-6d13-429f-8634-fefa562a1d30`::ref 1 aborting False >Thread-70150::INFO::2014-09-25 17:21:39,375::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70150::DEBUG::2014-09-25 17:21:39,375::task::1191::Storage.TaskManager.Task::(prepare) Task=`4d491209-6d13-429f-8634-fefa562a1d30`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70150::DEBUG::2014-09-25 17:21:39,375::task::595::Storage.TaskManager.Task::(_updateState) Task=`4d491209-6d13-429f-8634-fefa562a1d30`::moving from state preparing -> state finished >Thread-70150::DEBUG::2014-09-25 17:21:39,375::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70150::DEBUG::2014-09-25 17:21:39,376::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70150::DEBUG::2014-09-25 17:21:39,376::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70150::DEBUG::2014-09-25 17:21:39,376::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70150::DEBUG::2014-09-25 17:21:39,377::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70150::DEBUG::2014-09-25 17:21:39,377::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70150::DEBUG::2014-09-25 17:21:39,377::task::993::Storage.TaskManager.Task::(_decref) Task=`4d491209-6d13-429f-8634-fefa562a1d30`::ref 0 aborting False >Thread-70150::DEBUG::2014-09-25 17:21:39,378::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70150::DEBUG::2014-09-25 17:21:39,378::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:39,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:39,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0269951 s, 37.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:40,968::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:40,969::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70151::DEBUG::2014-09-25 17:21:40,971::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:41,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:41,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132769 s, 77.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:43,203::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:43,203::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:43,203::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:43,203::__init__::232::IOProcess::(_processLogs) (107915) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:43,206::__init__::232::IOProcess::(_processLogs) (107915) Queuing response >Thread-79::DEBUG::2014-09-25 17:21:43,208::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:21:43,227::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00404646 s, 139 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:43,228::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:43,228::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:43,229::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:43,229::__init__::232::IOProcess::(_processLogs) (107916) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:43,232::__init__::232::IOProcess::(_processLogs) (107916) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:43,233::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:43,234::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:43,234::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:43,234::__init__::232::IOProcess::(_processLogs) (107917) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:43,237::__init__::232::IOProcess::(_processLogs) (107917) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:43,237::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:43,238::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:43,238::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:43,238::__init__::232::IOProcess::(_processLogs) (107918) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:43,262::__init__::232::IOProcess::(_processLogs) (107918) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:43,263::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:43,263::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:43,263::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:43,263::__init__::232::IOProcess::(_processLogs) (107919) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:43,268::__init__::232::IOProcess::(_processLogs) (107919) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:43,371::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:43,372::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:43,372::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:43,372::__init__::232::IOProcess::(_processLogs) (107920) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:43,373::__init__::232::IOProcess::(_processLogs) (107920) Queuing response >Thread-80::DEBUG::2014-09-25 17:21:43,374::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:21:43,383::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000412191 s, 917 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:43,384::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:43,385::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:43,386::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:43,386::__init__::232::IOProcess::(_processLogs) (107921) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:43,387::__init__::232::IOProcess::(_processLogs) (107921) Queuing response >Dummy-148::DEBUG::2014-09-25 17:21:43,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:43,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141808 s, 72.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:43,983::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:43,985::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70152::DEBUG::2014-09-25 17:21:43,987::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:45,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:45,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0245114 s, 41.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:47,001::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:47,006::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70153::DEBUG::2014-09-25 17:21:47,011::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:47,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:48,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202006 s, 50.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:48,642::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:48,643::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:48,643::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:48,643::__init__::232::IOProcess::(_processLogs) (107922) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:48,647::__init__::232::IOProcess::(_processLogs) (107922) Queuing response >Thread-81::DEBUG::2014-09-25 17:21:48,648::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:21:48,660::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00260858 s, 142 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:48,661::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:48,662::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:48,662::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:48,662::__init__::232::IOProcess::(_processLogs) (107923) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:48,665::__init__::232::IOProcess::(_processLogs) (107923) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:49,476::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70154::DEBUG::2014-09-25 17:21:49,477::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:21:49,477::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70154::DEBUG::2014-09-25 17:21:49,480::task::595::Storage.TaskManager.Task::(_updateState) Task=`423d66e6-d0d8-4635-8fce-be11ac13e09d`::moving from state init -> state preparing >Thread-70154::INFO::2014-09-25 17:21:49,480::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70154::INFO::2014-09-25 17:21:49,481::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70154::DEBUG::2014-09-25 17:21:49,481::task::1191::Storage.TaskManager.Task::(prepare) Task=`423d66e6-d0d8-4635-8fce-be11ac13e09d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70154::DEBUG::2014-09-25 17:21:49,481::task::595::Storage.TaskManager.Task::(_updateState) Task=`423d66e6-d0d8-4635-8fce-be11ac13e09d`::moving from state preparing -> state finished >Thread-70154::DEBUG::2014-09-25 17:21:49,481::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70154::DEBUG::2014-09-25 17:21:49,481::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70154::DEBUG::2014-09-25 17:21:49,482::task::993::Storage.TaskManager.Task::(_decref) Task=`423d66e6-d0d8-4635-8fce-be11ac13e09d`::ref 0 aborting False >Thread-70154::DEBUG::2014-09-25 17:21:49,482::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70154::DEBUG::2014-09-25 17:21:49,482::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:49,493::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70155::DEBUG::2014-09-25 17:21:49,494::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:21:49,494::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70155::DEBUG::2014-09-25 17:21:49,496::task::595::Storage.TaskManager.Task::(_updateState) Task=`793f1c35-0f7f-4d2e-9bf3-5daaa8b51e00`::moving from state init -> state preparing >Thread-70155::INFO::2014-09-25 17:21:49,496::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70155::DEBUG::2014-09-25 17:21:49,497::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`88df92ad-e1c1-4e47-9b41-8fe6fbbe140b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70155::DEBUG::2014-09-25 17:21:49,497::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70155::DEBUG::2014-09-25 17:21:49,498::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70155::DEBUG::2014-09-25 17:21:49,498::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`88df92ad-e1c1-4e47-9b41-8fe6fbbe140b`::Granted request >Thread-70155::DEBUG::2014-09-25 17:21:49,498::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`793f1c35-0f7f-4d2e-9bf3-5daaa8b51e00`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70155::DEBUG::2014-09-25 17:21:49,499::task::993::Storage.TaskManager.Task::(_decref) Task=`793f1c35-0f7f-4d2e-9bf3-5daaa8b51e00`::ref 1 aborting False >Thread-70155::INFO::2014-09-25 17:21:49,503::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70155::DEBUG::2014-09-25 17:21:49,503::task::1191::Storage.TaskManager.Task::(prepare) Task=`793f1c35-0f7f-4d2e-9bf3-5daaa8b51e00`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70155::DEBUG::2014-09-25 17:21:49,504::task::595::Storage.TaskManager.Task::(_updateState) Task=`793f1c35-0f7f-4d2e-9bf3-5daaa8b51e00`::moving from state preparing -> state finished >Thread-70155::DEBUG::2014-09-25 17:21:49,504::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70155::DEBUG::2014-09-25 17:21:49,504::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70155::DEBUG::2014-09-25 17:21:49,505::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70155::DEBUG::2014-09-25 17:21:49,505::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70155::DEBUG::2014-09-25 17:21:49,505::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70155::DEBUG::2014-09-25 17:21:49,505::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70155::DEBUG::2014-09-25 17:21:49,506::task::993::Storage.TaskManager.Task::(_decref) Task=`793f1c35-0f7f-4d2e-9bf3-5daaa8b51e00`::ref 0 aborting False >Thread-70155::DEBUG::2014-09-25 17:21:49,506::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70155::DEBUG::2014-09-25 17:21:49,507::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:50,028::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:50,029::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70156::DEBUG::2014-09-25 17:21:50,031::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:50,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:50,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171034 s, 59.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:21:52,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:52,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130054 s, 78.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:53,047::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:53,048::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70157::DEBUG::2014-09-25 17:21:53,051::task::595::Storage.TaskManager.Task::(_updateState) Task=`b95a196e-bc6b-4ce8-bbe9-bb4a24fe3862`::moving from state init -> state preparing >Thread-70157::INFO::2014-09-25 17:21:53,051::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70157::INFO::2014-09-25 17:21:53,051::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00404646', 'lastCheck': '9.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000412191', 'lastCheck': '9.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00260858', 'lastCheck': '4.4', 'valid': True}} >Thread-70157::DEBUG::2014-09-25 17:21:53,052::task::1191::Storage.TaskManager.Task::(prepare) Task=`b95a196e-bc6b-4ce8-bbe9-bb4a24fe3862`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00404646', 'lastCheck': '9.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000412191', 'lastCheck': '9.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00260858', 'lastCheck': '4.4', 'valid': True}} >Thread-70157::DEBUG::2014-09-25 17:21:53,052::task::595::Storage.TaskManager.Task::(_updateState) Task=`b95a196e-bc6b-4ce8-bbe9-bb4a24fe3862`::moving from state preparing -> state finished >Thread-70157::DEBUG::2014-09-25 17:21:53,052::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70157::DEBUG::2014-09-25 17:21:53,052::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70157::DEBUG::2014-09-25 17:21:53,053::task::993::Storage.TaskManager.Task::(_decref) Task=`b95a196e-bc6b-4ce8-bbe9-bb4a24fe3862`::ref 0 aborting False >Thread-70157::DEBUG::2014-09-25 17:21:53,055::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:53,074::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:53,076::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70158::DEBUG::2014-09-25 17:21:53,077::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:21:53,270::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:53,270::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:53,270::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:53,271::__init__::232::IOProcess::(_processLogs) (107924) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:53,278::__init__::232::IOProcess::(_processLogs) (107924) Queuing response >Thread-79::DEBUG::2014-09-25 17:21:53,280::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:21:53,295::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00317323 s, 177 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:53,296::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:53,297::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:53,297::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:53,297::__init__::232::IOProcess::(_processLogs) (107925) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:53,300::__init__::232::IOProcess::(_processLogs) (107925) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:53,301::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:53,301::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:53,301::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:53,302::__init__::232::IOProcess::(_processLogs) (107926) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:53,302::__init__::232::IOProcess::(_processLogs) (107926) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:53,303::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:53,303::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:53,303::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:53,303::__init__::232::IOProcess::(_processLogs) (107927) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:53,304::__init__::232::IOProcess::(_processLogs) (107927) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:53,305::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:53,305::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:53,305::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:53,305::__init__::232::IOProcess::(_processLogs) (107928) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:21:53,306::__init__::232::IOProcess::(_processLogs) (107928) Queuing response >Thread-29::DEBUG::2014-09-25 17:21:53,387::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:53,387::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:53,387::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:53,387::__init__::232::IOProcess::(_processLogs) (107929) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:53,388::__init__::232::IOProcess::(_processLogs) (107929) Queuing response >Thread-80::DEBUG::2014-09-25 17:21:53,389::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:21:53,397::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000566904 s, 667 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:53,399::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:53,399::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:53,399::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:53,401::__init__::232::IOProcess::(_processLogs) (107930) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:53,401::__init__::232::IOProcess::(_processLogs) (107930) Queuing response >Dummy-148::DEBUG::2014-09-25 17:21:54,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:54,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0232694 s, 44.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:56,131::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:56,133::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70159::DEBUG::2014-09-25 17:21:56,135::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:21:56,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:56,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015819 s, 64.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:21:58,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:21:58,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0228055 s, 44.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:58,667::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:58,668::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:58,668::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:58,668::__init__::232::IOProcess::(_processLogs) (107931) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:58,685::__init__::232::IOProcess::(_processLogs) (107931) Queuing response >Thread-81::DEBUG::2014-09-25 17:21:58,686::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:21:58,710::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00232273 s, 160 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:21:58,712::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:21:58,712::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:21:58,712::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:21:58,712::__init__::232::IOProcess::(_processLogs) (107932) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:21:58,719::__init__::232::IOProcess::(_processLogs) (107932) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:59,148::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:59,150::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70160::DEBUG::2014-09-25 17:21:59,152::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:59,533::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70161::DEBUG::2014-09-25 17:21:59,534::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:21:59,534::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70161::DEBUG::2014-09-25 17:21:59,536::task::595::Storage.TaskManager.Task::(_updateState) Task=`ce030c8f-4f88-4da7-89d0-c638660a937e`::moving from state init -> state preparing >Thread-70161::INFO::2014-09-25 17:21:59,537::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70161::INFO::2014-09-25 17:21:59,537::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70161::DEBUG::2014-09-25 17:21:59,537::task::1191::Storage.TaskManager.Task::(prepare) Task=`ce030c8f-4f88-4da7-89d0-c638660a937e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70161::DEBUG::2014-09-25 17:21:59,538::task::595::Storage.TaskManager.Task::(_updateState) Task=`ce030c8f-4f88-4da7-89d0-c638660a937e`::moving from state preparing -> state finished >Thread-70161::DEBUG::2014-09-25 17:21:59,538::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70161::DEBUG::2014-09-25 17:21:59,538::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70161::DEBUG::2014-09-25 17:21:59,538::task::993::Storage.TaskManager.Task::(_decref) Task=`ce030c8f-4f88-4da7-89d0-c638660a937e`::ref 0 aborting False >Thread-70161::DEBUG::2014-09-25 17:21:59,539::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70161::DEBUG::2014-09-25 17:21:59,539::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:21:59,552::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:21:59,553::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70162::DEBUG::2014-09-25 17:21:59,553::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70162::DEBUG::2014-09-25 17:21:59,555::task::595::Storage.TaskManager.Task::(_updateState) Task=`ab80048e-ac1b-4b56-b114-13c70df4b093`::moving from state init -> state preparing >Thread-70162::INFO::2014-09-25 17:21:59,555::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70162::DEBUG::2014-09-25 17:21:59,556::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`88486fb2-216b-42d7-bf16-88ae983131d2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70162::DEBUG::2014-09-25 17:21:59,556::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70162::DEBUG::2014-09-25 17:21:59,557::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70162::DEBUG::2014-09-25 17:21:59,557::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`88486fb2-216b-42d7-bf16-88ae983131d2`::Granted request >Thread-70162::DEBUG::2014-09-25 17:21:59,558::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ab80048e-ac1b-4b56-b114-13c70df4b093`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70162::DEBUG::2014-09-25 17:21:59,558::task::993::Storage.TaskManager.Task::(_decref) Task=`ab80048e-ac1b-4b56-b114-13c70df4b093`::ref 1 aborting False >Thread-70162::INFO::2014-09-25 17:21:59,562::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70162::DEBUG::2014-09-25 17:21:59,562::task::1191::Storage.TaskManager.Task::(prepare) Task=`ab80048e-ac1b-4b56-b114-13c70df4b093`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70162::DEBUG::2014-09-25 17:21:59,563::task::595::Storage.TaskManager.Task::(_updateState) Task=`ab80048e-ac1b-4b56-b114-13c70df4b093`::moving from state preparing -> state finished >Thread-70162::DEBUG::2014-09-25 17:21:59,563::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70162::DEBUG::2014-09-25 17:21:59,563::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70162::DEBUG::2014-09-25 17:21:59,564::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70162::DEBUG::2014-09-25 17:21:59,564::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70162::DEBUG::2014-09-25 17:21:59,564::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70162::DEBUG::2014-09-25 17:21:59,564::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70162::DEBUG::2014-09-25 17:21:59,565::task::993::Storage.TaskManager.Task::(_decref) Task=`ab80048e-ac1b-4b56-b114-13c70df4b093`::ref 0 aborting False >Thread-70162::DEBUG::2014-09-25 17:21:59,565::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70162::DEBUG::2014-09-25 17:21:59,566::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:00,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:00,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0220668 s, 46.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:02,165::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:02,167::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70163::DEBUG::2014-09-25 17:22:02,169::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:02,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:02,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021398 s, 47.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:03,307::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:03,308::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:03,308::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:03,308::__init__::232::IOProcess::(_processLogs) (107933) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:03,309::__init__::232::IOProcess::(_processLogs) (107933) Queuing response >Thread-79::DEBUG::2014-09-25 17:22:03,310::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:22:03,322::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00148899 s, 377 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:03,323::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:03,323::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:03,324::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:03,324::__init__::232::IOProcess::(_processLogs) (107934) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:03,324::__init__::232::IOProcess::(_processLogs) (107934) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:03,326::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:03,327::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:03,327::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:03,327::__init__::232::IOProcess::(_processLogs) (107935) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:03,328::__init__::232::IOProcess::(_processLogs) (107935) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:03,328::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:03,329::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:03,329::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:03,330::__init__::232::IOProcess::(_processLogs) (107936) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:03,330::__init__::232::IOProcess::(_processLogs) (107936) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:03,330::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:03,330::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:03,331::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:03,332::__init__::232::IOProcess::(_processLogs) (107937) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:03,332::__init__::232::IOProcess::(_processLogs) (107937) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:03,402::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:03,402::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:03,402::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:03,402::__init__::232::IOProcess::(_processLogs) (107938) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:03,403::__init__::232::IOProcess::(_processLogs) (107938) Queuing response >Thread-80::DEBUG::2014-09-25 17:22:03,404::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:22:03,416::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00139683 s, 271 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:03,417::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:03,417::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:03,418::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:03,419::__init__::232::IOProcess::(_processLogs) (107939) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:03,419::__init__::232::IOProcess::(_processLogs) (107939) Queuing response >Dummy-148::DEBUG::2014-09-25 17:22:04,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:04,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0190303 s, 53.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:05,184::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:05,186::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70164::DEBUG::2014-09-25 17:22:05,188::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:06,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:06,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162529 s, 63.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:08,205::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:08,207::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70165::DEBUG::2014-09-25 17:22:08,210::task::595::Storage.TaskManager.Task::(_updateState) Task=`c6ce0d53-5cc0-44db-ab76-d7af73989044`::moving from state init -> state preparing >Thread-70165::INFO::2014-09-25 17:22:08,210::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70165::INFO::2014-09-25 17:22:08,210::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00148899', 'lastCheck': '4.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00139683', 'lastCheck': '4.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00232273', 'lastCheck': '9.5', 'valid': True}} >Thread-70165::DEBUG::2014-09-25 17:22:08,211::task::1191::Storage.TaskManager.Task::(prepare) Task=`c6ce0d53-5cc0-44db-ab76-d7af73989044`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00148899', 'lastCheck': '4.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00139683', 'lastCheck': '4.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00232273', 'lastCheck': '9.5', 'valid': True}} >Thread-70165::DEBUG::2014-09-25 17:22:08,211::task::595::Storage.TaskManager.Task::(_updateState) Task=`c6ce0d53-5cc0-44db-ab76-d7af73989044`::moving from state preparing -> state finished >Thread-70165::DEBUG::2014-09-25 17:22:08,211::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70165::DEBUG::2014-09-25 17:22:08,211::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70165::DEBUG::2014-09-25 17:22:08,212::task::993::Storage.TaskManager.Task::(_decref) Task=`c6ce0d53-5cc0-44db-ab76-d7af73989044`::ref 0 aborting False >Thread-70165::DEBUG::2014-09-25 17:22:08,214::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:08,238::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:08,239::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70166::DEBUG::2014-09-25 17:22:08,241::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:08,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:08,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169628 s, 60.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:08,721::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:08,721::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:08,722::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:08,722::__init__::232::IOProcess::(_processLogs) (107940) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:08,725::__init__::232::IOProcess::(_processLogs) (107940) Queuing response >Thread-81::DEBUG::2014-09-25 17:22:08,726::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:22:08,739::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00312569 s, 119 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:08,740::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:08,740::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:08,741::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:08,741::__init__::232::IOProcess::(_processLogs) (107941) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:08,743::__init__::232::IOProcess::(_processLogs) (107941) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:09,597::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:09,599::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70167::DEBUG::2014-09-25 17:22:09,599::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70167::DEBUG::2014-09-25 17:22:09,602::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6e6d897-2076-419e-9951-9c9918c65c49`::moving from state init -> state preparing >Thread-70167::INFO::2014-09-25 17:22:09,602::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70167::INFO::2014-09-25 17:22:09,603::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70167::DEBUG::2014-09-25 17:22:09,603::task::1191::Storage.TaskManager.Task::(prepare) Task=`f6e6d897-2076-419e-9951-9c9918c65c49`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70167::DEBUG::2014-09-25 17:22:09,603::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6e6d897-2076-419e-9951-9c9918c65c49`::moving from state preparing -> state finished >Thread-70167::DEBUG::2014-09-25 17:22:09,603::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70167::DEBUG::2014-09-25 17:22:09,604::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70167::DEBUG::2014-09-25 17:22:09,604::task::993::Storage.TaskManager.Task::(_decref) Task=`f6e6d897-2076-419e-9951-9c9918c65c49`::ref 0 aborting False >Thread-70167::DEBUG::2014-09-25 17:22:09,604::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70167::DEBUG::2014-09-25 17:22:09,604::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:09,624::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70168::DEBUG::2014-09-25 17:22:09,625::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:22:09,625::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70168::DEBUG::2014-09-25 17:22:09,627::task::595::Storage.TaskManager.Task::(_updateState) Task=`47d62fe4-240b-4c58-8239-73e10322b04d`::moving from state init -> state preparing >Thread-70168::INFO::2014-09-25 17:22:09,627::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70168::DEBUG::2014-09-25 17:22:09,628::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`abc30f4c-4b8b-4a77-9e26-17107955f543`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70168::DEBUG::2014-09-25 17:22:09,628::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70168::DEBUG::2014-09-25 17:22:09,629::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70168::DEBUG::2014-09-25 17:22:09,629::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`abc30f4c-4b8b-4a77-9e26-17107955f543`::Granted request >Thread-70168::DEBUG::2014-09-25 17:22:09,630::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`47d62fe4-240b-4c58-8239-73e10322b04d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70168::DEBUG::2014-09-25 17:22:09,630::task::993::Storage.TaskManager.Task::(_decref) Task=`47d62fe4-240b-4c58-8239-73e10322b04d`::ref 1 aborting False >Thread-70168::INFO::2014-09-25 17:22:09,634::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70168::DEBUG::2014-09-25 17:22:09,635::task::1191::Storage.TaskManager.Task::(prepare) Task=`47d62fe4-240b-4c58-8239-73e10322b04d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70168::DEBUG::2014-09-25 17:22:09,635::task::595::Storage.TaskManager.Task::(_updateState) Task=`47d62fe4-240b-4c58-8239-73e10322b04d`::moving from state preparing -> state finished >Thread-70168::DEBUG::2014-09-25 17:22:09,635::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70168::DEBUG::2014-09-25 17:22:09,635::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70168::DEBUG::2014-09-25 17:22:09,636::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70168::DEBUG::2014-09-25 17:22:09,636::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70168::DEBUG::2014-09-25 17:22:09,636::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70168::DEBUG::2014-09-25 17:22:09,636::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70168::DEBUG::2014-09-25 17:22:09,637::task::993::Storage.TaskManager.Task::(_decref) Task=`47d62fe4-240b-4c58-8239-73e10322b04d`::ref 0 aborting False >Thread-70168::DEBUG::2014-09-25 17:22:09,637::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70168::DEBUG::2014-09-25 17:22:09,638::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:10,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:10,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191677 s, 53.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:11,297::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:11,298::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70169::DEBUG::2014-09-25 17:22:11,301::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:12,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:12,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204505 s, 50.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:13,333::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:13,333::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:13,333::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:13,333::__init__::232::IOProcess::(_processLogs) (107942) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:13,337::__init__::232::IOProcess::(_processLogs) (107942) Queuing response >Thread-79::DEBUG::2014-09-25 17:22:13,338::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:22:13,355::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00419315 s, 134 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:13,356::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:13,357::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:13,357::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:13,357::__init__::232::IOProcess::(_processLogs) (107943) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:13,359::__init__::232::IOProcess::(_processLogs) (107943) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:13,360::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:13,360::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:13,361::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:13,361::__init__::232::IOProcess::(_processLogs) (107944) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:13,361::__init__::232::IOProcess::(_processLogs) (107944) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:13,362::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:13,362::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:13,363::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:13,363::__init__::232::IOProcess::(_processLogs) (107945) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:13,364::__init__::232::IOProcess::(_processLogs) (107945) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:13,364::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:13,365::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:13,365::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:13,366::__init__::232::IOProcess::(_processLogs) (107946) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:13,366::__init__::232::IOProcess::(_processLogs) (107946) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:13,420::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:13,420::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:13,420::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:13,421::__init__::232::IOProcess::(_processLogs) (107947) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:13,421::__init__::232::IOProcess::(_processLogs) (107947) Queuing response >Thread-80::DEBUG::2014-09-25 17:22:13,422::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:22:13,431::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000508622 s, 743 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:13,432::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:13,432::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:13,433::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:13,434::__init__::232::IOProcess::(_processLogs) (107948) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:13,434::__init__::232::IOProcess::(_processLogs) (107948) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:14,318::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:14,319::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70170::DEBUG::2014-09-25 17:22:14,321::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:14,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:14,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189073 s, 54.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:22:16,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:16,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0254633 s, 40.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:17,341::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:17,342::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70171::DEBUG::2014-09-25 17:22:17,344::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:22:18,746::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:18,746::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:18,746::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:18,746::__init__::232::IOProcess::(_processLogs) (107949) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:18,751::__init__::232::IOProcess::(_processLogs) (107949) Queuing response >Thread-81::DEBUG::2014-09-25 17:22:18,753::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:22:18,797::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0124162 s, 29.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:18,798::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:18,798::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:18,799::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:18,799::__init__::232::IOProcess::(_processLogs) (107950) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:18,810::__init__::232::IOProcess::(_processLogs) (107950) Queuing response >Dummy-148::DEBUG::2014-09-25 17:22:19,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:19,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205658 s, 49.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:19,673::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:19,674::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70172::DEBUG::2014-09-25 17:22:19,674::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70172::DEBUG::2014-09-25 17:22:19,676::task::595::Storage.TaskManager.Task::(_updateState) Task=`49e32707-c14d-47e4-bf93-a29451639b67`::moving from state init -> state preparing >Thread-70172::INFO::2014-09-25 17:22:19,677::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70172::INFO::2014-09-25 17:22:19,677::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70172::DEBUG::2014-09-25 17:22:19,677::task::1191::Storage.TaskManager.Task::(prepare) Task=`49e32707-c14d-47e4-bf93-a29451639b67`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70172::DEBUG::2014-09-25 17:22:19,678::task::595::Storage.TaskManager.Task::(_updateState) Task=`49e32707-c14d-47e4-bf93-a29451639b67`::moving from state preparing -> state finished >Thread-70172::DEBUG::2014-09-25 17:22:19,678::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70172::DEBUG::2014-09-25 17:22:19,678::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70172::DEBUG::2014-09-25 17:22:19,678::task::993::Storage.TaskManager.Task::(_decref) Task=`49e32707-c14d-47e4-bf93-a29451639b67`::ref 0 aborting False >Thread-70172::DEBUG::2014-09-25 17:22:19,679::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70172::DEBUG::2014-09-25 17:22:19,679::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:19,692::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:19,694::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70173::DEBUG::2014-09-25 17:22:19,694::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70173::DEBUG::2014-09-25 17:22:19,695::task::595::Storage.TaskManager.Task::(_updateState) Task=`afb4a0eb-c8c8-4b6e-9d8b-c760ac7e0c6f`::moving from state init -> state preparing >Thread-70173::INFO::2014-09-25 17:22:19,696::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70173::DEBUG::2014-09-25 17:22:19,696::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aa3ac8bb-a9b1-408d-aab7-114f1c8c2e32`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70173::DEBUG::2014-09-25 17:22:19,697::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70173::DEBUG::2014-09-25 17:22:19,697::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70173::DEBUG::2014-09-25 17:22:19,697::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aa3ac8bb-a9b1-408d-aab7-114f1c8c2e32`::Granted request >Thread-70173::DEBUG::2014-09-25 17:22:19,698::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`afb4a0eb-c8c8-4b6e-9d8b-c760ac7e0c6f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70173::DEBUG::2014-09-25 17:22:19,698::task::993::Storage.TaskManager.Task::(_decref) Task=`afb4a0eb-c8c8-4b6e-9d8b-c760ac7e0c6f`::ref 1 aborting False >Thread-70173::INFO::2014-09-25 17:22:19,703::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70173::DEBUG::2014-09-25 17:22:19,703::task::1191::Storage.TaskManager.Task::(prepare) Task=`afb4a0eb-c8c8-4b6e-9d8b-c760ac7e0c6f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70173::DEBUG::2014-09-25 17:22:19,703::task::595::Storage.TaskManager.Task::(_updateState) Task=`afb4a0eb-c8c8-4b6e-9d8b-c760ac7e0c6f`::moving from state preparing -> state finished >Thread-70173::DEBUG::2014-09-25 17:22:19,703::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70173::DEBUG::2014-09-25 17:22:19,704::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70173::DEBUG::2014-09-25 17:22:19,704::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70173::DEBUG::2014-09-25 17:22:19,704::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70173::DEBUG::2014-09-25 17:22:19,705::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70173::DEBUG::2014-09-25 17:22:19,705::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70173::DEBUG::2014-09-25 17:22:19,705::task::993::Storage.TaskManager.Task::(_decref) Task=`afb4a0eb-c8c8-4b6e-9d8b-c760ac7e0c6f`::ref 0 aborting False >Thread-70173::DEBUG::2014-09-25 17:22:19,706::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70173::DEBUG::2014-09-25 17:22:19,706::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:20,365::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:20,366::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70174::DEBUG::2014-09-25 17:22:20,368::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:21,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:21,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.259554 s, 3.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:22:23,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:22:23,368::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:23,368::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:23,369::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:23,369::__init__::232::IOProcess::(_processLogs) (107951) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:23,370::__init__::232::IOProcess::(_processLogs) (107951) Queuing response >Thread-79::DEBUG::2014-09-25 17:22:23,371::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:22:23,383::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000968729 s, 580 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:23,388::__init__::232::IOProcess::(_processLogs) Receiving request... >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:23,387::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Dummy-148::DEBUG::2014-09-25 17:22:23,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146505 s, 69.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:23,389::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:23,392::__init__::232::IOProcess::(_processLogs) Extracting request information... >JsonRpcServer::DEBUG::2014-09-25 17:22:23,394::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-29::DEBUG::2014-09-25 17:22:23,397::__init__::232::IOProcess::(_processLogs) (107952) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:23,402::__init__::232::IOProcess::(_processLogs) (107952) Queuing response >Thread-70175::DEBUG::2014-09-25 17:22:23,408::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb6f6a41-0217-4ad1-8b65-b18a387b6345`::moving from state init -> state preparing >Thread-29::DEBUG::2014-09-25 17:22:23,408::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-70175::INFO::2014-09-25 17:22:23,409::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29::DEBUG::2014-09-25 17:22:23,411::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-70175::INFO::2014-09-25 17:22:23,413::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00419315', 'lastCheck': '10.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000508622', 'lastCheck': '10.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0124162', 'lastCheck': '4.6', 'valid': True}} >Thread-70175::DEBUG::2014-09-25 17:22:23,415::task::1191::Storage.TaskManager.Task::(prepare) Task=`bb6f6a41-0217-4ad1-8b65-b18a387b6345`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00419315', 'lastCheck': '10.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000508622', 'lastCheck': '10.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0124162', 'lastCheck': '4.6', 'valid': True}} >Thread-29::DEBUG::2014-09-25 17:22:23,415::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-70175::DEBUG::2014-09-25 17:22:23,417::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb6f6a41-0217-4ad1-8b65-b18a387b6345`::moving from state preparing -> state finished >Thread-70175::DEBUG::2014-09-25 17:22:23,418::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29::DEBUG::2014-09-25 17:22:23,419::__init__::232::IOProcess::(_processLogs) (107953) Got request for method 'access' >Thread-70175::DEBUG::2014-09-25 17:22:23,419::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70175::DEBUG::2014-09-25 17:22:23,421::task::993::Storage.TaskManager.Task::(_decref) Task=`bb6f6a41-0217-4ad1-8b65-b18a387b6345`::ref 0 aborting False >Thread-29::DEBUG::2014-09-25 17:22:23,422::__init__::232::IOProcess::(_processLogs) (107953) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:23,429::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:23,433::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-70175::DEBUG::2014-09-25 17:22:23,436::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:22:23,437::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:23,441::__init__::232::IOProcess::(_processLogs) (107954) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:23,442::__init__::232::IOProcess::(_processLogs) (107954) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:23,445::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:23,447::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:23,448::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:23,450::__init__::232::IOProcess::(_processLogs) (107955) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:23,453::__init__::232::IOProcess::(_processLogs) (107955) Queuing response >Thread-80::DEBUG::2014-09-25 17:22:23,454::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:22:23,463::__init__::232::IOProcess::(_processLogs) Receiving request... >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:23,466::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-29::DEBUG::2014-09-25 17:22:23,466::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-80::DEBUG::2014-09-25 17:22:23,467::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000427815 s, 884 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:23,470::__init__::232::IOProcess::(_processLogs) Extracting request information... >JsonRpcServer::DEBUG::2014-09-25 17:22:23,473::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-29::DEBUG::2014-09-25 17:22:23,474::__init__::232::IOProcess::(_processLogs) (107956) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:23,476::__init__::232::IOProcess::(_processLogs) (107956) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:23,480::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-70176::DEBUG::2014-09-25 17:22:23,482::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:22:23,483::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:23,485::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:23,487::__init__::232::IOProcess::(_processLogs) (107957) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:23,488::__init__::232::IOProcess::(_processLogs) (107957) Queuing response >Dummy-148::DEBUG::2014-09-25 17:22:25,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:25,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0247712 s, 41.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:26,556::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:26,558::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70177::DEBUG::2014-09-25 17:22:26,560::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:27,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:27,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212985 s, 48.1 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 17:22:28,811::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:22:28,816::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:28,816::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:28,816::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:28,817::__init__::232::IOProcess::(_processLogs) (107958) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:22:28,817::__init__::232::IOProcess::(_processLogs) (107958) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:28,817::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:28,819::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:28,821::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:28,823::__init__::232::IOProcess::(_processLogs) (107959) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:22:28,824::__init__::232::IOProcess::(_processLogs) (107959) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:28,826::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:28,827::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:28,828::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:28,830::__init__::232::IOProcess::(_processLogs) (107960) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:22:28,832::__init__::232::IOProcess::(_processLogs) (107960) Queuing response >Thread-81::DEBUG::2014-09-25 17:22:28,834::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:22:28,835::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:28,835::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:28,835::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:28,835::__init__::232::IOProcess::(_processLogs) (107961) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:22:28,855::__init__::232::IOProcess::(_processLogs) (107961) Queuing response >Thread-81::DEBUG::2014-09-25 17:22:28,855::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:22:28,856::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:28,857::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:28,857::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:28,857::__init__::232::IOProcess::(_processLogs) (107962) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:22:28,858::__init__::232::IOProcess::(_processLogs) (107962) Queuing response >Thread-81::DEBUG::2014-09-25 17:22:28,858::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:22:28,859::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:28,860::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:28,861::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:28,861::__init__::232::IOProcess::(_processLogs) (107963) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:28,861::__init__::232::IOProcess::(_processLogs) (107963) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:28,861::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:28,862::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:28,862::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:28,862::__init__::232::IOProcess::(_processLogs) (107964) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:22:28,863::__init__::232::IOProcess::(_processLogs) (107964) Queuing response >Thread-81::DEBUG::2014-09-25 17:22:28,863::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 17:22:28,864::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:22:28,864::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:22:28,865::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:28,865::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:28,865::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:28,865::__init__::232::IOProcess::(_processLogs) (107965) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:28,866::__init__::232::IOProcess::(_processLogs) (107965) Queuing response >Thread-81::DEBUG::2014-09-25 17:22:28,867::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:22:28,884::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00754507 s, 49.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:28,885::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:28,885::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:28,886::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:28,886::__init__::232::IOProcess::(_processLogs) (107966) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:28,892::__init__::232::IOProcess::(_processLogs) (107966) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:29,576::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:29,577::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70181::DEBUG::2014-09-25 17:22:29,580::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:29,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:29,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132917 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:29,739::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70182::DEBUG::2014-09-25 17:22:29,741::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:22:29,741::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70182::DEBUG::2014-09-25 17:22:29,744::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ec3770b-a0c1-4d5f-9a65-22d90ab8bb34`::moving from state init -> state preparing >Thread-70182::INFO::2014-09-25 17:22:29,744::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70182::INFO::2014-09-25 17:22:29,745::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70182::DEBUG::2014-09-25 17:22:29,745::task::1191::Storage.TaskManager.Task::(prepare) Task=`0ec3770b-a0c1-4d5f-9a65-22d90ab8bb34`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70182::DEBUG::2014-09-25 17:22:29,745::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ec3770b-a0c1-4d5f-9a65-22d90ab8bb34`::moving from state preparing -> state finished >Thread-70182::DEBUG::2014-09-25 17:22:29,745::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70182::DEBUG::2014-09-25 17:22:29,745::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70182::DEBUG::2014-09-25 17:22:29,746::task::993::Storage.TaskManager.Task::(_decref) Task=`0ec3770b-a0c1-4d5f-9a65-22d90ab8bb34`::ref 0 aborting False >Thread-70182::DEBUG::2014-09-25 17:22:29,746::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70182::DEBUG::2014-09-25 17:22:29,746::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:29,759::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:29,760::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70183::DEBUG::2014-09-25 17:22:29,760::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70183::DEBUG::2014-09-25 17:22:29,762::task::595::Storage.TaskManager.Task::(_updateState) Task=`e65c44d9-0616-46f0-a5ef-d25fbedc0cf7`::moving from state init -> state preparing >Thread-70183::INFO::2014-09-25 17:22:29,763::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70183::DEBUG::2014-09-25 17:22:29,763::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e698f918-d373-4c62-9c5c-aabfb0f1527f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70183::DEBUG::2014-09-25 17:22:29,763::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70183::DEBUG::2014-09-25 17:22:29,764::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70183::DEBUG::2014-09-25 17:22:29,764::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e698f918-d373-4c62-9c5c-aabfb0f1527f`::Granted request >Thread-70183::DEBUG::2014-09-25 17:22:29,765::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e65c44d9-0616-46f0-a5ef-d25fbedc0cf7`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70183::DEBUG::2014-09-25 17:22:29,765::task::993::Storage.TaskManager.Task::(_decref) Task=`e65c44d9-0616-46f0-a5ef-d25fbedc0cf7`::ref 1 aborting False >Thread-70183::INFO::2014-09-25 17:22:29,769::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70183::DEBUG::2014-09-25 17:22:29,770::task::1191::Storage.TaskManager.Task::(prepare) Task=`e65c44d9-0616-46f0-a5ef-d25fbedc0cf7`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70183::DEBUG::2014-09-25 17:22:29,770::task::595::Storage.TaskManager.Task::(_updateState) Task=`e65c44d9-0616-46f0-a5ef-d25fbedc0cf7`::moving from state preparing -> state finished >Thread-70183::DEBUG::2014-09-25 17:22:29,770::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70183::DEBUG::2014-09-25 17:22:29,770::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70183::DEBUG::2014-09-25 17:22:29,771::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70183::DEBUG::2014-09-25 17:22:29,771::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70183::DEBUG::2014-09-25 17:22:29,772::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70183::DEBUG::2014-09-25 17:22:29,772::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70183::DEBUG::2014-09-25 17:22:29,773::task::993::Storage.TaskManager.Task::(_decref) Task=`e65c44d9-0616-46f0-a5ef-d25fbedc0cf7`::ref 0 aborting False >Thread-70183::DEBUG::2014-09-25 17:22:29,774::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70183::DEBUG::2014-09-25 17:22:29,775::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:31,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:31,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154907 s, 66.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:32,596::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:32,598::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70184::DEBUG::2014-09-25 17:22:32,600::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:22:33,482::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:33,482::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:33,482::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:33,483::__init__::232::IOProcess::(_processLogs) (107967) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:33,486::__init__::232::IOProcess::(_processLogs) (107967) Queuing response >Thread-79::DEBUG::2014-09-25 17:22:33,488::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:22:33,496::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:33,497::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:33,497::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:33,497::__init__::232::IOProcess::(_processLogs) (107968) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:33,498::__init__::232::IOProcess::(_processLogs) (107968) Queuing response >Thread-80::DEBUG::2014-09-25 17:22:33,499::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:22:33,508::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00398913 s, 141 kB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 17:22:33,508::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000379374 s, 996 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:33,511::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:33,511::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:33,512::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:33,512::__init__::232::IOProcess::(_processLogs) (107969) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:33,512::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:33,512::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:33,513::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:33,513::__init__::232::IOProcess::(_processLogs) (107970) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:33,514::__init__::232::IOProcess::(_processLogs) (107970) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:33,515::__init__::232::IOProcess::(_processLogs) (107969) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:33,517::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:33,517::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:33,518::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:33,518::__init__::232::IOProcess::(_processLogs) (107971) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:33,518::__init__::232::IOProcess::(_processLogs) (107971) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:33,519::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:33,519::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:33,520::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:33,520::__init__::232::IOProcess::(_processLogs) (107972) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:33,520::__init__::232::IOProcess::(_processLogs) (107972) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:33,521::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:33,521::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:33,521::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:33,522::__init__::232::IOProcess::(_processLogs) (107973) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:33,522::__init__::232::IOProcess::(_processLogs) (107973) Queuing response >Dummy-148::DEBUG::2014-09-25 17:22:33,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:33,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215632 s, 47.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:35,617::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:35,619::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70185::DEBUG::2014-09-25 17:22:35,621::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:35,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:35,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200124 s, 51.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:22:37,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:37,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136743 s, 74.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:38,637::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:38,638::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70186::DEBUG::2014-09-25 17:22:38,641::task::595::Storage.TaskManager.Task::(_updateState) Task=`7a007888-10bc-459e-a82b-ec521c16ebf2`::moving from state init -> state preparing >Thread-70186::INFO::2014-09-25 17:22:38,641::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70186::INFO::2014-09-25 17:22:38,642::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00398913', 'lastCheck': '5.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000379374', 'lastCheck': '5.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00754507', 'lastCheck': '9.7', 'valid': True}} >Thread-70186::DEBUG::2014-09-25 17:22:38,642::task::1191::Storage.TaskManager.Task::(prepare) Task=`7a007888-10bc-459e-a82b-ec521c16ebf2`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00398913', 'lastCheck': '5.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000379374', 'lastCheck': '5.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00754507', 'lastCheck': '9.7', 'valid': True}} >Thread-70186::DEBUG::2014-09-25 17:22:38,642::task::595::Storage.TaskManager.Task::(_updateState) Task=`7a007888-10bc-459e-a82b-ec521c16ebf2`::moving from state preparing -> state finished >Thread-70186::DEBUG::2014-09-25 17:22:38,642::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70186::DEBUG::2014-09-25 17:22:38,643::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70186::DEBUG::2014-09-25 17:22:38,643::task::993::Storage.TaskManager.Task::(_decref) Task=`7a007888-10bc-459e-a82b-ec521c16ebf2`::ref 0 aborting False >Thread-70186::DEBUG::2014-09-25 17:22:38,645::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:38,667::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:38,668::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70187::DEBUG::2014-09-25 17:22:38,670::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:22:38,894::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:38,894::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:38,894::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:38,895::__init__::232::IOProcess::(_processLogs) (107974) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:38,921::__init__::232::IOProcess::(_processLogs) (107974) Queuing response >Thread-81::DEBUG::2014-09-25 17:22:38,922::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:22:38,937::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00567329 s, 65.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:38,939::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:38,939::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:38,939::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:38,940::__init__::232::IOProcess::(_processLogs) (107975) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:38,940::__init__::232::IOProcess::(_processLogs) (107975) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:39,805::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:39,806::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70188::DEBUG::2014-09-25 17:22:39,806::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70188::DEBUG::2014-09-25 17:22:39,808::task::595::Storage.TaskManager.Task::(_updateState) Task=`a4c08ea0-44c3-4a0e-a8ca-7418379919bb`::moving from state init -> state preparing >Thread-70188::INFO::2014-09-25 17:22:39,809::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70188::INFO::2014-09-25 17:22:39,809::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70188::DEBUG::2014-09-25 17:22:39,810::task::1191::Storage.TaskManager.Task::(prepare) Task=`a4c08ea0-44c3-4a0e-a8ca-7418379919bb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70188::DEBUG::2014-09-25 17:22:39,810::task::595::Storage.TaskManager.Task::(_updateState) Task=`a4c08ea0-44c3-4a0e-a8ca-7418379919bb`::moving from state preparing -> state finished >Thread-70188::DEBUG::2014-09-25 17:22:39,810::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70188::DEBUG::2014-09-25 17:22:39,810::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70188::DEBUG::2014-09-25 17:22:39,810::task::993::Storage.TaskManager.Task::(_decref) Task=`a4c08ea0-44c3-4a0e-a8ca-7418379919bb`::ref 0 aborting False >Thread-70188::DEBUG::2014-09-25 17:22:39,811::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70188::DEBUG::2014-09-25 17:22:39,811::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:39,829::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:39,830::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70189::DEBUG::2014-09-25 17:22:39,830::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70189::DEBUG::2014-09-25 17:22:39,832::task::595::Storage.TaskManager.Task::(_updateState) Task=`83245b44-81b1-497d-be90-738c2a5dfe54`::moving from state init -> state preparing >Thread-70189::INFO::2014-09-25 17:22:39,832::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70189::DEBUG::2014-09-25 17:22:39,833::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aa86f57d-216d-4433-b4e5-a8d267e6867e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70189::DEBUG::2014-09-25 17:22:39,833::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70189::DEBUG::2014-09-25 17:22:39,834::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70189::DEBUG::2014-09-25 17:22:39,834::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aa86f57d-216d-4433-b4e5-a8d267e6867e`::Granted request >Thread-70189::DEBUG::2014-09-25 17:22:39,834::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`83245b44-81b1-497d-be90-738c2a5dfe54`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70189::DEBUG::2014-09-25 17:22:39,835::task::993::Storage.TaskManager.Task::(_decref) Task=`83245b44-81b1-497d-be90-738c2a5dfe54`::ref 1 aborting False >Thread-70189::INFO::2014-09-25 17:22:39,839::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70189::DEBUG::2014-09-25 17:22:39,839::task::1191::Storage.TaskManager.Task::(prepare) Task=`83245b44-81b1-497d-be90-738c2a5dfe54`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70189::DEBUG::2014-09-25 17:22:39,840::task::595::Storage.TaskManager.Task::(_updateState) Task=`83245b44-81b1-497d-be90-738c2a5dfe54`::moving from state preparing -> state finished >Thread-70189::DEBUG::2014-09-25 17:22:39,840::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70189::DEBUG::2014-09-25 17:22:39,840::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70189::DEBUG::2014-09-25 17:22:39,841::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70189::DEBUG::2014-09-25 17:22:39,841::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70189::DEBUG::2014-09-25 17:22:39,841::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70189::DEBUG::2014-09-25 17:22:39,841::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70189::DEBUG::2014-09-25 17:22:39,842::task::993::Storage.TaskManager.Task::(_decref) Task=`83245b44-81b1-497d-be90-738c2a5dfe54`::ref 0 aborting False >Thread-70189::DEBUG::2014-09-25 17:22:39,842::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70189::DEBUG::2014-09-25 17:22:39,843::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:39,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:39,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202706 s, 50.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:41,728::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:41,730::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70190::DEBUG::2014-09-25 17:22:41,732::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:41,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:41,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193609 s, 52.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:43,516::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:43,516::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:43,516::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:43,517::__init__::232::IOProcess::(_processLogs) (107976) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:43,517::__init__::232::IOProcess::(_processLogs) (107976) Queuing response >Thread-80::DEBUG::2014-09-25 17:22:43,518::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:22:43,526::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:43,527::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:43,527::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:43,527::__init__::232::IOProcess::(_processLogs) (107977) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:43,529::__init__::232::IOProcess::(_processLogs) (107977) Queuing response >Thread-80::DEBUG::2014-09-25 17:22:43,530::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000489096 s, 773 kB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 17:22:43,531::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:22:43,533::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:43,541::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:43,541::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:43,541::__init__::232::IOProcess::(_processLogs) (107978) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:43,542::__init__::232::IOProcess::(_processLogs) (107978) Queuing response >Thread-79::DEBUG::2014-09-25 17:22:43,552::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00598532 s, 93.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:43,553::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:43,553::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:43,554::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:43,554::__init__::232::IOProcess::(_processLogs) (107979) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:43,554::__init__::232::IOProcess::(_processLogs) (107979) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:43,556::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:43,556::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:43,556::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:43,556::__init__::232::IOProcess::(_processLogs) (107980) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:43,560::__init__::232::IOProcess::(_processLogs) (107980) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:43,561::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:43,561::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:43,561::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:43,562::__init__::232::IOProcess::(_processLogs) (107981) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:43,562::__init__::232::IOProcess::(_processLogs) (107981) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:43,563::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:43,564::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:43,564::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:43,564::__init__::232::IOProcess::(_processLogs) (107982) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:43,567::__init__::232::IOProcess::(_processLogs) (107982) Queuing response >Dummy-148::DEBUG::2014-09-25 17:22:43,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:44,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210126 s, 48.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:44,750::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:44,751::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70191::DEBUG::2014-09-25 17:22:44,754::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:46,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:46,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0308034 s, 33.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:47,770::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:47,772::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70192::DEBUG::2014-09-25 17:22:47,776::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:48,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:48,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0251977 s, 40.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:48,942::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:48,942::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:48,943::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:48,944::__init__::232::IOProcess::(_processLogs) (107983) Got request for method 'statvfs' >Thread-81::DEBUG::2014-09-25 17:22:48,944::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:22:48,945::__init__::232::IOProcess::(_processLogs) (107983) Queuing response >Thread-81::DEBUG::2014-09-25 17:22:48,954::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000506184 s, 733 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:48,955::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:48,956::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:48,956::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:48,956::__init__::232::IOProcess::(_processLogs) (107984) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:48,957::__init__::232::IOProcess::(_processLogs) (107984) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:49,872::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70193::DEBUG::2014-09-25 17:22:49,874::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:22:49,874::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70193::DEBUG::2014-09-25 17:22:49,876::task::595::Storage.TaskManager.Task::(_updateState) Task=`910d30d8-4be3-4628-a874-bae5e6035c5b`::moving from state init -> state preparing >Thread-70193::INFO::2014-09-25 17:22:49,876::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70193::INFO::2014-09-25 17:22:49,877::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70193::DEBUG::2014-09-25 17:22:49,877::task::1191::Storage.TaskManager.Task::(prepare) Task=`910d30d8-4be3-4628-a874-bae5e6035c5b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70193::DEBUG::2014-09-25 17:22:49,877::task::595::Storage.TaskManager.Task::(_updateState) Task=`910d30d8-4be3-4628-a874-bae5e6035c5b`::moving from state preparing -> state finished >Thread-70193::DEBUG::2014-09-25 17:22:49,877::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70193::DEBUG::2014-09-25 17:22:49,878::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70193::DEBUG::2014-09-25 17:22:49,878::task::993::Storage.TaskManager.Task::(_decref) Task=`910d30d8-4be3-4628-a874-bae5e6035c5b`::ref 0 aborting False >Thread-70193::DEBUG::2014-09-25 17:22:49,878::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70193::DEBUG::2014-09-25 17:22:49,879::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:49,891::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70194::DEBUG::2014-09-25 17:22:49,893::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:22:49,893::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70194::DEBUG::2014-09-25 17:22:49,895::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f05250a-c8e7-4650-9039-6d21a30448ed`::moving from state init -> state preparing >Thread-70194::INFO::2014-09-25 17:22:49,895::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70194::DEBUG::2014-09-25 17:22:49,896::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`bfab2414-e2ab-4d61-924e-52513b1bf80c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70194::DEBUG::2014-09-25 17:22:49,896::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70194::DEBUG::2014-09-25 17:22:49,897::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70194::DEBUG::2014-09-25 17:22:49,897::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`bfab2414-e2ab-4d61-924e-52513b1bf80c`::Granted request >Thread-70194::DEBUG::2014-09-25 17:22:49,897::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`8f05250a-c8e7-4650-9039-6d21a30448ed`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70194::DEBUG::2014-09-25 17:22:49,898::task::993::Storage.TaskManager.Task::(_decref) Task=`8f05250a-c8e7-4650-9039-6d21a30448ed`::ref 1 aborting False >Thread-70194::INFO::2014-09-25 17:22:49,902::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70194::DEBUG::2014-09-25 17:22:49,902::task::1191::Storage.TaskManager.Task::(prepare) Task=`8f05250a-c8e7-4650-9039-6d21a30448ed`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70194::DEBUG::2014-09-25 17:22:49,902::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f05250a-c8e7-4650-9039-6d21a30448ed`::moving from state preparing -> state finished >Thread-70194::DEBUG::2014-09-25 17:22:49,903::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70194::DEBUG::2014-09-25 17:22:49,903::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70194::DEBUG::2014-09-25 17:22:49,903::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70194::DEBUG::2014-09-25 17:22:49,904::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70194::DEBUG::2014-09-25 17:22:49,904::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70194::DEBUG::2014-09-25 17:22:49,904::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70194::DEBUG::2014-09-25 17:22:49,905::task::993::Storage.TaskManager.Task::(_decref) Task=`8f05250a-c8e7-4650-9039-6d21a30448ed`::ref 0 aborting False >Thread-70194::DEBUG::2014-09-25 17:22:49,905::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70194::DEBUG::2014-09-25 17:22:49,906::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:50,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:50,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133122 s, 76.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:50,790::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:50,792::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70195::DEBUG::2014-09-25 17:22:50,794::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:52,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:52,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156108 s, 65.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:53,544::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:53,544::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:53,545::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:53,545::__init__::232::IOProcess::(_processLogs) (107985) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:53,546::__init__::232::IOProcess::(_processLogs) (107985) Queuing response >Thread-80::DEBUG::2014-09-25 17:22:53,547::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:22:53,555::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000475755 s, 795 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:53,557::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:53,557::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:53,557::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:53,558::__init__::232::IOProcess::(_processLogs) (107986) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:53,558::__init__::232::IOProcess::(_processLogs) (107986) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:53,568::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:53,568::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:53,568::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:53,569::__init__::232::IOProcess::(_processLogs) (107987) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:53,569::__init__::232::IOProcess::(_processLogs) (107987) Queuing response >Thread-79::DEBUG::2014-09-25 17:22:53,570::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:22:53,578::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0004258 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:53,579::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:53,580::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:53,580::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:53,580::__init__::232::IOProcess::(_processLogs) (107988) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:53,581::__init__::232::IOProcess::(_processLogs) (107988) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:53,582::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:53,583::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:53,583::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:53,584::__init__::232::IOProcess::(_processLogs) (107989) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:53,584::__init__::232::IOProcess::(_processLogs) (107989) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:53,584::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:53,585::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:53,585::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:53,586::__init__::232::IOProcess::(_processLogs) (107990) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:53,586::__init__::232::IOProcess::(_processLogs) (107990) Queuing response >Thread-29::DEBUG::2014-09-25 17:22:53,586::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:53,587::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:53,588::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:53,588::__init__::232::IOProcess::(_processLogs) (107991) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:22:53,588::__init__::232::IOProcess::(_processLogs) (107991) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:53,805::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:53,806::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70196::DEBUG::2014-09-25 17:22:53,809::task::595::Storage.TaskManager.Task::(_updateState) Task=`518635b6-c7dd-4858-ad0e-21c4ba6588ed`::moving from state init -> state preparing >Thread-70196::INFO::2014-09-25 17:22:53,809::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70196::INFO::2014-09-25 17:22:53,810::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0004258', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000475755', 'lastCheck': '0.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000506184', 'lastCheck': '4.9', 'valid': True}} >Thread-70196::DEBUG::2014-09-25 17:22:53,810::task::1191::Storage.TaskManager.Task::(prepare) Task=`518635b6-c7dd-4858-ad0e-21c4ba6588ed`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0004258', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000475755', 'lastCheck': '0.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000506184', 'lastCheck': '4.9', 'valid': True}} >Thread-70196::DEBUG::2014-09-25 17:22:53,810::task::595::Storage.TaskManager.Task::(_updateState) Task=`518635b6-c7dd-4858-ad0e-21c4ba6588ed`::moving from state preparing -> state finished >Thread-70196::DEBUG::2014-09-25 17:22:53,810::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70196::DEBUG::2014-09-25 17:22:53,811::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70196::DEBUG::2014-09-25 17:22:53,811::task::993::Storage.TaskManager.Task::(_decref) Task=`518635b6-c7dd-4858-ad0e-21c4ba6588ed`::ref 0 aborting False >Thread-70196::DEBUG::2014-09-25 17:22:53,813::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:53,827::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:53,829::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70197::DEBUG::2014-09-25 17:22:53,830::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:54,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:54,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135471 s, 75.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:22:56,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:56,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0247051 s, 41.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:56,873::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:56,875::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70198::DEBUG::2014-09-25 17:22:56,877::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:22:58,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:22:58,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207701 s, 49.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:58,959::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:58,959::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:58,960::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:58,960::__init__::232::IOProcess::(_processLogs) (107992) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:58,973::__init__::232::IOProcess::(_processLogs) (107992) Queuing response >Thread-81::DEBUG::2014-09-25 17:22:58,974::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:22:59,010::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00281751 s, 132 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:22:59,011::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:22:59,011::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:22:59,012::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:22:59,012::__init__::232::IOProcess::(_processLogs) (107993) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:22:59,017::__init__::232::IOProcess::(_processLogs) (107993) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:59,888::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:22:59,890::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70199::DEBUG::2014-09-25 17:22:59,892::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:59,927::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70200::DEBUG::2014-09-25 17:22:59,928::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:22:59,928::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70200::DEBUG::2014-09-25 17:22:59,930::task::595::Storage.TaskManager.Task::(_updateState) Task=`e399c120-dd23-49e7-a418-7c3e36edbab8`::moving from state init -> state preparing >Thread-70200::INFO::2014-09-25 17:22:59,931::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70200::INFO::2014-09-25 17:22:59,931::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70200::DEBUG::2014-09-25 17:22:59,931::task::1191::Storage.TaskManager.Task::(prepare) Task=`e399c120-dd23-49e7-a418-7c3e36edbab8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70200::DEBUG::2014-09-25 17:22:59,932::task::595::Storage.TaskManager.Task::(_updateState) Task=`e399c120-dd23-49e7-a418-7c3e36edbab8`::moving from state preparing -> state finished >Thread-70200::DEBUG::2014-09-25 17:22:59,932::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70200::DEBUG::2014-09-25 17:22:59,932::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70200::DEBUG::2014-09-25 17:22:59,932::task::993::Storage.TaskManager.Task::(_decref) Task=`e399c120-dd23-49e7-a418-7c3e36edbab8`::ref 0 aborting False >Thread-70200::DEBUG::2014-09-25 17:22:59,933::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70200::DEBUG::2014-09-25 17:22:59,933::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:22:59,944::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70201::DEBUG::2014-09-25 17:22:59,945::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:22:59,945::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70201::DEBUG::2014-09-25 17:22:59,947::task::595::Storage.TaskManager.Task::(_updateState) Task=`cf2f0d67-eb6d-4786-990c-9670f8bb8ff2`::moving from state init -> state preparing >Thread-70201::INFO::2014-09-25 17:22:59,947::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70201::DEBUG::2014-09-25 17:22:59,948::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`21e45bb0-64c9-4620-9e06-76f692e29519`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70201::DEBUG::2014-09-25 17:22:59,948::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70201::DEBUG::2014-09-25 17:22:59,949::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70201::DEBUG::2014-09-25 17:22:59,949::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`21e45bb0-64c9-4620-9e06-76f692e29519`::Granted request >Thread-70201::DEBUG::2014-09-25 17:22:59,950::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`cf2f0d67-eb6d-4786-990c-9670f8bb8ff2`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70201::DEBUG::2014-09-25 17:22:59,950::task::993::Storage.TaskManager.Task::(_decref) Task=`cf2f0d67-eb6d-4786-990c-9670f8bb8ff2`::ref 1 aborting False >Thread-70201::INFO::2014-09-25 17:22:59,954::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70201::DEBUG::2014-09-25 17:22:59,955::task::1191::Storage.TaskManager.Task::(prepare) Task=`cf2f0d67-eb6d-4786-990c-9670f8bb8ff2`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70201::DEBUG::2014-09-25 17:22:59,955::task::595::Storage.TaskManager.Task::(_updateState) Task=`cf2f0d67-eb6d-4786-990c-9670f8bb8ff2`::moving from state preparing -> state finished >Thread-70201::DEBUG::2014-09-25 17:22:59,955::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70201::DEBUG::2014-09-25 17:22:59,955::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70201::DEBUG::2014-09-25 17:22:59,956::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70201::DEBUG::2014-09-25 17:22:59,956::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70201::DEBUG::2014-09-25 17:22:59,956::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70201::DEBUG::2014-09-25 17:22:59,956::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70201::DEBUG::2014-09-25 17:22:59,957::task::993::Storage.TaskManager.Task::(_decref) Task=`cf2f0d67-eb6d-4786-990c-9670f8bb8ff2`::ref 0 aborting False >Thread-70201::DEBUG::2014-09-25 17:22:59,957::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70201::DEBUG::2014-09-25 17:22:59,958::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:00,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:00,586::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130259 s, 78.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:23:02,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:02,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147674 s, 69.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:02,909::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:02,911::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70202::DEBUG::2014-09-25 17:23:02,913::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:23:03,560::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:03,561::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:03,561::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:03,561::__init__::232::IOProcess::(_processLogs) (107994) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:03,561::__init__::232::IOProcess::(_processLogs) (107994) Queuing response >Thread-80::DEBUG::2014-09-25 17:23:03,563::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:23:03,571::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000454391 s, 832 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:03,573::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:03,573::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:03,573::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:03,573::__init__::232::IOProcess::(_processLogs) (107995) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:03,574::__init__::232::IOProcess::(_processLogs) (107995) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:03,588::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:03,588::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:03,589::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:03,589::__init__::232::IOProcess::(_processLogs) (107996) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:03,595::__init__::232::IOProcess::(_processLogs) (107996) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:03,596::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:23:03,614::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00245294 s, 229 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:03,615::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:03,616::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:03,616::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:03,616::__init__::232::IOProcess::(_processLogs) (107997) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:03,620::__init__::232::IOProcess::(_processLogs) (107997) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:03,621::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:03,622::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:03,623::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:03,623::__init__::232::IOProcess::(_processLogs) (107998) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:03,623::__init__::232::IOProcess::(_processLogs) (107998) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:03,623::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:03,624::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:03,625::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:03,625::__init__::232::IOProcess::(_processLogs) (107999) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:03,625::__init__::232::IOProcess::(_processLogs) (107999) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:03,626::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:03,626::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:03,626::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:03,627::__init__::232::IOProcess::(_processLogs) (108000) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:03,628::__init__::232::IOProcess::(_processLogs) (108000) Queuing response >Dummy-148::DEBUG::2014-09-25 17:23:04,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:04,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150009 s, 68.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:05,928::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:05,930::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70203::DEBUG::2014-09-25 17:23:05,932::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:06,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:06,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209675 s, 48.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:23:08,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:08,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133548 s, 76.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:08,946::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:08,947::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70204::DEBUG::2014-09-25 17:23:08,950::task::595::Storage.TaskManager.Task::(_updateState) Task=`514a410a-6a2e-46f5-abf7-752f9e4cf401`::moving from state init -> state preparing >Thread-70204::INFO::2014-09-25 17:23:08,950::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70204::INFO::2014-09-25 17:23:08,950::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00245294', 'lastCheck': '5.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000454391', 'lastCheck': '5.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00281751', 'lastCheck': '9.9', 'valid': True}} >Thread-70204::DEBUG::2014-09-25 17:23:08,951::task::1191::Storage.TaskManager.Task::(prepare) Task=`514a410a-6a2e-46f5-abf7-752f9e4cf401`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00245294', 'lastCheck': '5.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000454391', 'lastCheck': '5.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00281751', 'lastCheck': '9.9', 'valid': True}} >Thread-70204::DEBUG::2014-09-25 17:23:08,951::task::595::Storage.TaskManager.Task::(_updateState) Task=`514a410a-6a2e-46f5-abf7-752f9e4cf401`::moving from state preparing -> state finished >Thread-70204::DEBUG::2014-09-25 17:23:08,951::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70204::DEBUG::2014-09-25 17:23:08,951::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70204::DEBUG::2014-09-25 17:23:08,952::task::993::Storage.TaskManager.Task::(_decref) Task=`514a410a-6a2e-46f5-abf7-752f9e4cf401`::ref 0 aborting False >Thread-70204::DEBUG::2014-09-25 17:23:08,954::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:08,969::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:08,971::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70205::DEBUG::2014-09-25 17:23:08,973::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:23:09,019::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:09,019::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:09,019::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:09,020::__init__::232::IOProcess::(_processLogs) (108001) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:09,024::__init__::232::IOProcess::(_processLogs) (108001) Queuing response >Thread-81::DEBUG::2014-09-25 17:23:09,024::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:23:09,045::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00574132 s, 64.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:09,046::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:09,047::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:09,047::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:09,047::__init__::232::IOProcess::(_processLogs) (108002) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:09,049::__init__::232::IOProcess::(_processLogs) (108002) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:09,983::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70206::DEBUG::2014-09-25 17:23:09,984::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:23:09,984::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70206::DEBUG::2014-09-25 17:23:09,986::task::595::Storage.TaskManager.Task::(_updateState) Task=`a011d1fc-987e-42df-b806-8b118cf82ba1`::moving from state init -> state preparing >Thread-70206::INFO::2014-09-25 17:23:09,987::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70206::INFO::2014-09-25 17:23:09,987::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70206::DEBUG::2014-09-25 17:23:09,988::task::1191::Storage.TaskManager.Task::(prepare) Task=`a011d1fc-987e-42df-b806-8b118cf82ba1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70206::DEBUG::2014-09-25 17:23:09,988::task::595::Storage.TaskManager.Task::(_updateState) Task=`a011d1fc-987e-42df-b806-8b118cf82ba1`::moving from state preparing -> state finished >Thread-70206::DEBUG::2014-09-25 17:23:09,988::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70206::DEBUG::2014-09-25 17:23:09,988::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70206::DEBUG::2014-09-25 17:23:09,989::task::993::Storage.TaskManager.Task::(_decref) Task=`a011d1fc-987e-42df-b806-8b118cf82ba1`::ref 0 aborting False >Thread-70206::DEBUG::2014-09-25 17:23:09,989::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70206::DEBUG::2014-09-25 17:23:09,989::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:09,999::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70207::DEBUG::2014-09-25 17:23:10,001::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:23:10,001::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70207::DEBUG::2014-09-25 17:23:10,003::task::595::Storage.TaskManager.Task::(_updateState) Task=`f31b49c9-b400-4ad0-b07a-8138d0ff6fba`::moving from state init -> state preparing >Thread-70207::INFO::2014-09-25 17:23:10,003::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70207::DEBUG::2014-09-25 17:23:10,004::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`435fedca-fc6a-49dd-94d2-cb27ceb702b8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70207::DEBUG::2014-09-25 17:23:10,004::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70207::DEBUG::2014-09-25 17:23:10,005::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70207::DEBUG::2014-09-25 17:23:10,005::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`435fedca-fc6a-49dd-94d2-cb27ceb702b8`::Granted request >Thread-70207::DEBUG::2014-09-25 17:23:10,005::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f31b49c9-b400-4ad0-b07a-8138d0ff6fba`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70207::DEBUG::2014-09-25 17:23:10,006::task::993::Storage.TaskManager.Task::(_decref) Task=`f31b49c9-b400-4ad0-b07a-8138d0ff6fba`::ref 1 aborting False >Thread-70207::INFO::2014-09-25 17:23:10,010::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70207::DEBUG::2014-09-25 17:23:10,010::task::1191::Storage.TaskManager.Task::(prepare) Task=`f31b49c9-b400-4ad0-b07a-8138d0ff6fba`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70207::DEBUG::2014-09-25 17:23:10,011::task::595::Storage.TaskManager.Task::(_updateState) Task=`f31b49c9-b400-4ad0-b07a-8138d0ff6fba`::moving from state preparing -> state finished >Thread-70207::DEBUG::2014-09-25 17:23:10,011::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70207::DEBUG::2014-09-25 17:23:10,011::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70207::DEBUG::2014-09-25 17:23:10,011::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70207::DEBUG::2014-09-25 17:23:10,012::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70207::DEBUG::2014-09-25 17:23:10,012::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70207::DEBUG::2014-09-25 17:23:10,012::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70207::DEBUG::2014-09-25 17:23:10,012::task::993::Storage.TaskManager.Task::(_decref) Task=`f31b49c9-b400-4ad0-b07a-8138d0ff6fba`::ref 0 aborting False >Thread-70207::DEBUG::2014-09-25 17:23:10,013::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70207::DEBUG::2014-09-25 17:23:10,013::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:10,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:10,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203315 s, 50.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:12,012::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:12,014::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70208::DEBUG::2014-09-25 17:23:12,016::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:12,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:12,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186759 s, 54.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:13,576::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:13,577::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:13,577::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:13,577::__init__::232::IOProcess::(_processLogs) (108003) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:13,578::__init__::232::IOProcess::(_processLogs) (108003) Queuing response >Thread-80::DEBUG::2014-09-25 17:23:13,579::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:23:13,587::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000406499 s, 930 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:13,589::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:13,589::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:13,590::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:13,591::__init__::232::IOProcess::(_processLogs) (108004) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:13,591::__init__::232::IOProcess::(_processLogs) (108004) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:13,628::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:13,628::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:13,628::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:13,629::__init__::232::IOProcess::(_processLogs) (108005) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:13,648::__init__::232::IOProcess::(_processLogs) (108005) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:13,649::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:23:13,677::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00899697 s, 62.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:13,678::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:13,679::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:13,679::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:13,679::__init__::232::IOProcess::(_processLogs) (108006) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:13,682::__init__::232::IOProcess::(_processLogs) (108006) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:13,683::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:13,683::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:13,684::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:13,684::__init__::232::IOProcess::(_processLogs) (108007) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:13,685::__init__::232::IOProcess::(_processLogs) (108007) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:13,685::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:13,686::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:13,686::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:13,687::__init__::232::IOProcess::(_processLogs) (108008) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:13,687::__init__::232::IOProcess::(_processLogs) (108008) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:13,687::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:13,688::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:13,688::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:13,688::__init__::232::IOProcess::(_processLogs) (108009) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:13,689::__init__::232::IOProcess::(_processLogs) (108009) Queuing response >Dummy-148::DEBUG::2014-09-25 17:23:14,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:14,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214422 s, 47.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:15,027::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:15,029::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70209::DEBUG::2014-09-25 17:23:15,032::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:16,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:16,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178013 s, 57.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:18,044::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:18,045::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70210::DEBUG::2014-09-25 17:23:18,047::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:19,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:23:19,052::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:19,054::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:19,055::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:19,056::__init__::232::IOProcess::(_processLogs) (108010) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:19,058::__init__::232::IOProcess::(_processLogs) (108010) Queuing response >Thread-81::DEBUG::2014-09-25 17:23:19,060::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:19,070::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0331019 s, 30.9 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 17:23:19,082::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000807954 s, 459 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:19,084::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:19,085::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:19,086::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:19,087::__init__::232::IOProcess::(_processLogs) (108011) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:19,088::__init__::232::IOProcess::(_processLogs) (108011) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:20,038::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:20,040::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70211::DEBUG::2014-09-25 17:23:20,040::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70211::DEBUG::2014-09-25 17:23:20,042::task::595::Storage.TaskManager.Task::(_updateState) Task=`fa223226-bf92-4fb3-888d-7e5a3aad2b27`::moving from state init -> state preparing >Thread-70211::INFO::2014-09-25 17:23:20,043::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70211::INFO::2014-09-25 17:23:20,043::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70211::DEBUG::2014-09-25 17:23:20,043::task::1191::Storage.TaskManager.Task::(prepare) Task=`fa223226-bf92-4fb3-888d-7e5a3aad2b27`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70211::DEBUG::2014-09-25 17:23:20,044::task::595::Storage.TaskManager.Task::(_updateState) Task=`fa223226-bf92-4fb3-888d-7e5a3aad2b27`::moving from state preparing -> state finished >Thread-70211::DEBUG::2014-09-25 17:23:20,044::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70211::DEBUG::2014-09-25 17:23:20,044::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70211::DEBUG::2014-09-25 17:23:20,044::task::993::Storage.TaskManager.Task::(_decref) Task=`fa223226-bf92-4fb3-888d-7e5a3aad2b27`::ref 0 aborting False >Thread-70211::DEBUG::2014-09-25 17:23:20,045::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70211::DEBUG::2014-09-25 17:23:20,045::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:20,055::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:20,057::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70212::DEBUG::2014-09-25 17:23:20,057::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70212::DEBUG::2014-09-25 17:23:20,058::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f508aea-843f-4db5-8922-63c9c3ae9607`::moving from state init -> state preparing >Thread-70212::INFO::2014-09-25 17:23:20,059::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70212::DEBUG::2014-09-25 17:23:20,059::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9a1acded-eed4-43c2-ba62-ea7a06770d64`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70212::DEBUG::2014-09-25 17:23:20,060::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70212::DEBUG::2014-09-25 17:23:20,060::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70212::DEBUG::2014-09-25 17:23:20,060::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9a1acded-eed4-43c2-ba62-ea7a06770d64`::Granted request >Thread-70212::DEBUG::2014-09-25 17:23:20,061::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9f508aea-843f-4db5-8922-63c9c3ae9607`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70212::DEBUG::2014-09-25 17:23:20,061::task::993::Storage.TaskManager.Task::(_decref) Task=`9f508aea-843f-4db5-8922-63c9c3ae9607`::ref 1 aborting False >Thread-70212::INFO::2014-09-25 17:23:20,066::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70212::DEBUG::2014-09-25 17:23:20,066::task::1191::Storage.TaskManager.Task::(prepare) Task=`9f508aea-843f-4db5-8922-63c9c3ae9607`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70212::DEBUG::2014-09-25 17:23:20,066::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f508aea-843f-4db5-8922-63c9c3ae9607`::moving from state preparing -> state finished >Thread-70212::DEBUG::2014-09-25 17:23:20,066::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70212::DEBUG::2014-09-25 17:23:20,067::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70212::DEBUG::2014-09-25 17:23:20,067::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70212::DEBUG::2014-09-25 17:23:20,067::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70212::DEBUG::2014-09-25 17:23:20,067::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70212::DEBUG::2014-09-25 17:23:20,068::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70212::DEBUG::2014-09-25 17:23:20,068::task::993::Storage.TaskManager.Task::(_decref) Task=`9f508aea-843f-4db5-8922-63c9c3ae9607`::ref 0 aborting False >Thread-70212::DEBUG::2014-09-25 17:23:20,068::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70212::DEBUG::2014-09-25 17:23:20,069::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:21,059::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:21,060::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70213::DEBUG::2014-09-25 17:23:21,062::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:21,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:21,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0196863 s, 52.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:23:23,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:23,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166666 s, 61.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:23,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:23,592::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:23,593::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:23,593::__init__::232::IOProcess::(_processLogs) (108012) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:23,593::__init__::232::IOProcess::(_processLogs) (108012) Queuing response >Thread-80::DEBUG::2014-09-25 17:23:23,594::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:23:23,603::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000443392 s, 853 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:23,604::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:23,605::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:23,605::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:23,605::__init__::232::IOProcess::(_processLogs) (108013) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:23,605::__init__::232::IOProcess::(_processLogs) (108013) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:23,690::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:23,690::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:23,691::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:23,691::__init__::232::IOProcess::(_processLogs) (108014) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:23,693::__init__::232::IOProcess::(_processLogs) (108014) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:23,694::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:23:23,707::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00434749 s, 129 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:23,708::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:23,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:23,709::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:23,709::__init__::232::IOProcess::(_processLogs) (108015) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:23,712::__init__::232::IOProcess::(_processLogs) (108015) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:23,713::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:23,713::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:23,714::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:23,714::__init__::232::IOProcess::(_processLogs) (108016) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:23,715::__init__::232::IOProcess::(_processLogs) (108016) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:23,715::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:23,715::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:23,716::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:23,716::__init__::232::IOProcess::(_processLogs) (108017) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:23,716::__init__::232::IOProcess::(_processLogs) (108017) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:23,717::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:23,718::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:23,719::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:23,719::__init__::232::IOProcess::(_processLogs) (108018) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:23,719::__init__::232::IOProcess::(_processLogs) (108018) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:24,075::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:24,076::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70214::DEBUG::2014-09-25 17:23:24,079::task::595::Storage.TaskManager.Task::(_updateState) Task=`47e24687-d248-4423-b5a3-bac664907249`::moving from state init -> state preparing >Thread-70214::INFO::2014-09-25 17:23:24,079::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70214::INFO::2014-09-25 17:23:24,080::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00434749', 'lastCheck': '0.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443392', 'lastCheck': '0.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000807954', 'lastCheck': '5.0', 'valid': True}} >Thread-70214::DEBUG::2014-09-25 17:23:24,080::task::1191::Storage.TaskManager.Task::(prepare) Task=`47e24687-d248-4423-b5a3-bac664907249`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00434749', 'lastCheck': '0.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443392', 'lastCheck': '0.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000807954', 'lastCheck': '5.0', 'valid': True}} >Thread-70214::DEBUG::2014-09-25 17:23:24,080::task::595::Storage.TaskManager.Task::(_updateState) Task=`47e24687-d248-4423-b5a3-bac664907249`::moving from state preparing -> state finished >Thread-70214::DEBUG::2014-09-25 17:23:24,080::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70214::DEBUG::2014-09-25 17:23:24,081::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70214::DEBUG::2014-09-25 17:23:24,081::task::993::Storage.TaskManager.Task::(_decref) Task=`47e24687-d248-4423-b5a3-bac664907249`::ref 0 aborting False >Thread-70214::DEBUG::2014-09-25 17:23:24,083::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:24,103::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:24,104::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70215::DEBUG::2014-09-25 17:23:24,106::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:25,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:25,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013057 s, 78.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:27,171::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:27,172::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70216::DEBUG::2014-09-25 17:23:27,174::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:27,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:27,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213842 s, 47.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:29,091::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:29,091::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:29,091::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:29,092::__init__::232::IOProcess::(_processLogs) (108019) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:29,096::__init__::232::IOProcess::(_processLogs) (108019) Queuing response >Thread-81::DEBUG::2014-09-25 17:23:29,097::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:23:29,109::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00150144 s, 247 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:29,110::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:29,110::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:29,110::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:29,111::__init__::232::IOProcess::(_processLogs) (108020) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:29,112::__init__::232::IOProcess::(_processLogs) (108020) Queuing response >Dummy-148::DEBUG::2014-09-25 17:23:29,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:29,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019635 s, 52.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:30,095::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70217::DEBUG::2014-09-25 17:23:30,096::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:23:30,096::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70217::DEBUG::2014-09-25 17:23:30,099::task::595::Storage.TaskManager.Task::(_updateState) Task=`a21935ed-c677-40f5-898c-c8a9a222adc3`::moving from state init -> state preparing >Thread-70217::INFO::2014-09-25 17:23:30,099::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70217::INFO::2014-09-25 17:23:30,100::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70217::DEBUG::2014-09-25 17:23:30,100::task::1191::Storage.TaskManager.Task::(prepare) Task=`a21935ed-c677-40f5-898c-c8a9a222adc3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70217::DEBUG::2014-09-25 17:23:30,100::task::595::Storage.TaskManager.Task::(_updateState) Task=`a21935ed-c677-40f5-898c-c8a9a222adc3`::moving from state preparing -> state finished >Thread-70217::DEBUG::2014-09-25 17:23:30,101::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70217::DEBUG::2014-09-25 17:23:30,101::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70217::DEBUG::2014-09-25 17:23:30,101::task::993::Storage.TaskManager.Task::(_decref) Task=`a21935ed-c677-40f5-898c-c8a9a222adc3`::ref 0 aborting False >Thread-70217::DEBUG::2014-09-25 17:23:30,101::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70217::DEBUG::2014-09-25 17:23:30,102::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:30,116::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:30,117::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70218::DEBUG::2014-09-25 17:23:30,117::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70218::DEBUG::2014-09-25 17:23:30,119::task::595::Storage.TaskManager.Task::(_updateState) Task=`69efa074-ec3e-4c17-ae90-bcf327f01769`::moving from state init -> state preparing >Thread-70218::INFO::2014-09-25 17:23:30,119::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70218::DEBUG::2014-09-25 17:23:30,120::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`da65374e-231c-4392-812b-f2643f8c4839`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70218::DEBUG::2014-09-25 17:23:30,120::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70218::DEBUG::2014-09-25 17:23:30,121::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70218::DEBUG::2014-09-25 17:23:30,121::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`da65374e-231c-4392-812b-f2643f8c4839`::Granted request >Thread-70218::DEBUG::2014-09-25 17:23:30,121::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`69efa074-ec3e-4c17-ae90-bcf327f01769`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70218::DEBUG::2014-09-25 17:23:30,122::task::993::Storage.TaskManager.Task::(_decref) Task=`69efa074-ec3e-4c17-ae90-bcf327f01769`::ref 1 aborting False >Thread-70218::INFO::2014-09-25 17:23:30,126::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70218::DEBUG::2014-09-25 17:23:30,126::task::1191::Storage.TaskManager.Task::(prepare) Task=`69efa074-ec3e-4c17-ae90-bcf327f01769`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70218::DEBUG::2014-09-25 17:23:30,126::task::595::Storage.TaskManager.Task::(_updateState) Task=`69efa074-ec3e-4c17-ae90-bcf327f01769`::moving from state preparing -> state finished >Thread-70218::DEBUG::2014-09-25 17:23:30,127::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70218::DEBUG::2014-09-25 17:23:30,127::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70218::DEBUG::2014-09-25 17:23:30,127::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70218::DEBUG::2014-09-25 17:23:30,128::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70218::DEBUG::2014-09-25 17:23:30,128::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70218::DEBUG::2014-09-25 17:23:30,128::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70218::DEBUG::2014-09-25 17:23:30,128::task::993::Storage.TaskManager.Task::(_decref) Task=`69efa074-ec3e-4c17-ae90-bcf327f01769`::ref 0 aborting False >Thread-70218::DEBUG::2014-09-25 17:23:30,129::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70218::DEBUG::2014-09-25 17:23:30,129::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:30,191::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:30,192::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70219::DEBUG::2014-09-25 17:23:30,194::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:31,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:31,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177173 s, 57.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:33,207::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:33,209::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70220::DEBUG::2014-09-25 17:23:33,211::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:33,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:33,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0287096 s, 35.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:33,608::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:33,608::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:33,608::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:33,608::__init__::232::IOProcess::(_processLogs) (108021) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:33,609::__init__::232::IOProcess::(_processLogs) (108021) Queuing response >Thread-80::DEBUG::2014-09-25 17:23:33,610::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:23:33,619::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000505209 s, 748 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:33,620::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:33,620::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:33,621::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:33,621::__init__::232::IOProcess::(_processLogs) (108022) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:33,621::__init__::232::IOProcess::(_processLogs) (108022) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:33,719::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:33,719::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:33,720::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:33,720::__init__::232::IOProcess::(_processLogs) (108023) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 17:23:33,721::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:23:33,722::__init__::232::IOProcess::(_processLogs) (108023) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:33,730::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00045491 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:33,731::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:33,731::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:33,732::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:33,732::__init__::232::IOProcess::(_processLogs) (108024) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:33,733::__init__::232::IOProcess::(_processLogs) (108024) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:33,734::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:33,734::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:33,734::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:33,734::__init__::232::IOProcess::(_processLogs) (108025) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:33,735::__init__::232::IOProcess::(_processLogs) (108025) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:33,736::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:33,736::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:33,737::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:33,737::__init__::232::IOProcess::(_processLogs) (108026) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:33,737::__init__::232::IOProcess::(_processLogs) (108026) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:33,738::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:33,738::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:33,739::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:33,739::__init__::232::IOProcess::(_processLogs) (108027) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:33,739::__init__::232::IOProcess::(_processLogs) (108027) Queuing response >Dummy-148::DEBUG::2014-09-25 17:23:35,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:35,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135048 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:36,223::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:36,225::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70221::DEBUG::2014-09-25 17:23:36,227::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:37,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:37,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191966 s, 53.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:39,113::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:39,114::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:39,114::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:39,114::__init__::232::IOProcess::(_processLogs) (108028) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:39,118::__init__::232::IOProcess::(_processLogs) (108028) Queuing response >Thread-81::DEBUG::2014-09-25 17:23:39,119::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:23:39,131::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00103944 s, 357 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:39,132::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:39,132::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:39,132::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:39,133::__init__::232::IOProcess::(_processLogs) (108029) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:39,138::__init__::232::IOProcess::(_processLogs) (108029) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:39,238::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:39,240::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70222::DEBUG::2014-09-25 17:23:39,242::task::595::Storage.TaskManager.Task::(_updateState) Task=`4744c87c-b99e-470a-9e67-664a9e4b2889`::moving from state init -> state preparing >Thread-70222::INFO::2014-09-25 17:23:39,243::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70222::INFO::2014-09-25 17:23:39,243::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00045491', 'lastCheck': '5.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000505209', 'lastCheck': '5.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00103944', 'lastCheck': '0.1', 'valid': True}} >Thread-70222::DEBUG::2014-09-25 17:23:39,243::task::1191::Storage.TaskManager.Task::(prepare) Task=`4744c87c-b99e-470a-9e67-664a9e4b2889`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00045491', 'lastCheck': '5.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000505209', 'lastCheck': '5.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00103944', 'lastCheck': '0.1', 'valid': True}} >Thread-70222::DEBUG::2014-09-25 17:23:39,244::task::595::Storage.TaskManager.Task::(_updateState) Task=`4744c87c-b99e-470a-9e67-664a9e4b2889`::moving from state preparing -> state finished >Thread-70222::DEBUG::2014-09-25 17:23:39,244::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70222::DEBUG::2014-09-25 17:23:39,244::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70222::DEBUG::2014-09-25 17:23:39,244::task::993::Storage.TaskManager.Task::(_decref) Task=`4744c87c-b99e-470a-9e67-664a9e4b2889`::ref 0 aborting False >Thread-70222::DEBUG::2014-09-25 17:23:39,247::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:39,262::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:39,264::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70223::DEBUG::2014-09-25 17:23:39,266::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:39,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:39,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0231337 s, 44.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:40,159::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70224::DEBUG::2014-09-25 17:23:40,160::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:23:40,160::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70224::DEBUG::2014-09-25 17:23:40,163::task::595::Storage.TaskManager.Task::(_updateState) Task=`a06bd183-562b-4e8d-8a64-91f9b70924be`::moving from state init -> state preparing >Thread-70224::INFO::2014-09-25 17:23:40,163::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70224::INFO::2014-09-25 17:23:40,164::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70224::DEBUG::2014-09-25 17:23:40,164::task::1191::Storage.TaskManager.Task::(prepare) Task=`a06bd183-562b-4e8d-8a64-91f9b70924be`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70224::DEBUG::2014-09-25 17:23:40,164::task::595::Storage.TaskManager.Task::(_updateState) Task=`a06bd183-562b-4e8d-8a64-91f9b70924be`::moving from state preparing -> state finished >Thread-70224::DEBUG::2014-09-25 17:23:40,164::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70224::DEBUG::2014-09-25 17:23:40,165::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70224::DEBUG::2014-09-25 17:23:40,165::task::993::Storage.TaskManager.Task::(_decref) Task=`a06bd183-562b-4e8d-8a64-91f9b70924be`::ref 0 aborting False >Thread-70224::DEBUG::2014-09-25 17:23:40,165::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70224::DEBUG::2014-09-25 17:23:40,166::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:40,176::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70225::DEBUG::2014-09-25 17:23:40,177::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:23:40,177::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70225::DEBUG::2014-09-25 17:23:40,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`ec815c79-dd0c-4ddb-8f99-63bd011fa7c2`::moving from state init -> state preparing >Thread-70225::INFO::2014-09-25 17:23:40,179::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70225::DEBUG::2014-09-25 17:23:40,180::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7f349206-96f2-4312-abd6-fbe18ed6aaf7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70225::DEBUG::2014-09-25 17:23:40,180::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70225::DEBUG::2014-09-25 17:23:40,181::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70225::DEBUG::2014-09-25 17:23:40,181::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7f349206-96f2-4312-abd6-fbe18ed6aaf7`::Granted request >Thread-70225::DEBUG::2014-09-25 17:23:40,182::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ec815c79-dd0c-4ddb-8f99-63bd011fa7c2`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70225::DEBUG::2014-09-25 17:23:40,182::task::993::Storage.TaskManager.Task::(_decref) Task=`ec815c79-dd0c-4ddb-8f99-63bd011fa7c2`::ref 1 aborting False >Thread-70225::INFO::2014-09-25 17:23:40,186::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70225::DEBUG::2014-09-25 17:23:40,186::task::1191::Storage.TaskManager.Task::(prepare) Task=`ec815c79-dd0c-4ddb-8f99-63bd011fa7c2`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70225::DEBUG::2014-09-25 17:23:40,187::task::595::Storage.TaskManager.Task::(_updateState) Task=`ec815c79-dd0c-4ddb-8f99-63bd011fa7c2`::moving from state preparing -> state finished >Thread-70225::DEBUG::2014-09-25 17:23:40,187::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70225::DEBUG::2014-09-25 17:23:40,187::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70225::DEBUG::2014-09-25 17:23:40,188::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70225::DEBUG::2014-09-25 17:23:40,188::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70225::DEBUG::2014-09-25 17:23:40,188::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70225::DEBUG::2014-09-25 17:23:40,188::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70225::DEBUG::2014-09-25 17:23:40,189::task::993::Storage.TaskManager.Task::(_decref) Task=`ec815c79-dd0c-4ddb-8f99-63bd011fa7c2`::ref 0 aborting False >Thread-70225::DEBUG::2014-09-25 17:23:40,189::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70225::DEBUG::2014-09-25 17:23:40,190::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:41,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:41,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0271038 s, 37.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:42,310::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:42,312::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70226::DEBUG::2014-09-25 17:23:42,314::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:23:43,623::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,623::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,623::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,624::__init__::232::IOProcess::(_processLogs) (108030) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:43,624::__init__::232::IOProcess::(_processLogs) (108030) Queuing response >Thread-80::DEBUG::2014-09-25 17:23:43,625::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:23:43,634::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000355685 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:43,635::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,635::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,636::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,636::__init__::232::IOProcess::(_processLogs) (108031) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:43,636::__init__::232::IOProcess::(_processLogs) (108031) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:43,739::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:23:43,743::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,744::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,744::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,744::__init__::232::IOProcess::(_processLogs) (108032) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:23:43,744::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,745::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,745::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,745::__init__::232::IOProcess::(_processLogs) (108033) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:23:43,746::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,747::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,747::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,747::__init__::232::IOProcess::(_processLogs) (108034) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:23:43,747::__init__::232::IOProcess::(_processLogs) (108032) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:43,748::__init__::232::IOProcess::(_processLogs) (108033) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:43,748::__init__::232::IOProcess::(_processLogs) (108034) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:43,753::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:23:43,753::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,754::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,754::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,754::__init__::232::IOProcess::(_processLogs) (108035) Got request for method 'touch' >Dummy-148::DEBUG::2014-09-25 17:23:43,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:23:43,799::__init__::232::IOProcess::(_processLogs) (108035) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:43,800::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:23:43,802::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,802::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,802::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,802::__init__::232::IOProcess::(_processLogs) (108036) Got request for method 'readfile' >Dummy-148::DEBUG::2014-09-25 17:23:43,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194918 s, 52.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:43,820::__init__::232::IOProcess::(_processLogs) (108036) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:43,820::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:23:43,824::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,824::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,825::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,826::__init__::232::IOProcess::(_processLogs) (108037) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:43,827::__init__::232::IOProcess::(_processLogs) (108037) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:43,829::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,830::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,831::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,832::__init__::232::IOProcess::(_processLogs) (108038) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:23:43,833::__init__::232::IOProcess::(_processLogs) (108038) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:43,834::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 17:23:43,835::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:23:43,836::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:23:43,837::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,838::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,839::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,840::__init__::232::IOProcess::(_processLogs) (108039) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:43,842::__init__::232::IOProcess::(_processLogs) (108039) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:43,843::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:23:43,863::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00793829 s, 70.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:43,866::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,867::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,868::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,868::__init__::232::IOProcess::(_processLogs) (108040) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:43,872::__init__::232::IOProcess::(_processLogs) (108040) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:43,874::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,875::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,876::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,877::__init__::232::IOProcess::(_processLogs) (108041) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:43,878::__init__::232::IOProcess::(_processLogs) (108041) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:43,880::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,881::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,883::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,883::__init__::232::IOProcess::(_processLogs) (108042) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:43,885::__init__::232::IOProcess::(_processLogs) (108042) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:43,887::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:43,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:43,889::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:43,889::__init__::232::IOProcess::(_processLogs) (108043) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:43,891::__init__::232::IOProcess::(_processLogs) (108043) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:45,329::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:45,331::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70230::DEBUG::2014-09-25 17:23:45,333::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:45,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:45,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185076 s, 55.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:23:47,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:47,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181338 s, 56.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:48,347::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:48,348::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70231::DEBUG::2014-09-25 17:23:48,351::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:23:49,140::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:49,140::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:49,141::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:49,141::__init__::232::IOProcess::(_processLogs) (108044) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:49,141::__init__::232::IOProcess::(_processLogs) (108044) Queuing response >Thread-81::DEBUG::2014-09-25 17:23:49,143::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:23:49,151::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000403655 s, 919 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:49,152::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:49,153::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:49,153::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:49,153::__init__::232::IOProcess::(_processLogs) (108045) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:49,154::__init__::232::IOProcess::(_processLogs) (108045) Queuing response >Dummy-148::DEBUG::2014-09-25 17:23:49,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:50,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177448 s, 57.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:50,214::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:50,216::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70232::DEBUG::2014-09-25 17:23:50,216::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70232::DEBUG::2014-09-25 17:23:50,218::task::595::Storage.TaskManager.Task::(_updateState) Task=`196d5d12-fe2d-4f85-b68c-38bdc7232b12`::moving from state init -> state preparing >Thread-70232::INFO::2014-09-25 17:23:50,218::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70232::INFO::2014-09-25 17:23:50,219::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70232::DEBUG::2014-09-25 17:23:50,219::task::1191::Storage.TaskManager.Task::(prepare) Task=`196d5d12-fe2d-4f85-b68c-38bdc7232b12`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70232::DEBUG::2014-09-25 17:23:50,219::task::595::Storage.TaskManager.Task::(_updateState) Task=`196d5d12-fe2d-4f85-b68c-38bdc7232b12`::moving from state preparing -> state finished >Thread-70232::DEBUG::2014-09-25 17:23:50,220::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70232::DEBUG::2014-09-25 17:23:50,220::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70232::DEBUG::2014-09-25 17:23:50,220::task::993::Storage.TaskManager.Task::(_decref) Task=`196d5d12-fe2d-4f85-b68c-38bdc7232b12`::ref 0 aborting False >Thread-70232::DEBUG::2014-09-25 17:23:50,220::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70232::DEBUG::2014-09-25 17:23:50,221::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:50,231::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70233::DEBUG::2014-09-25 17:23:50,233::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:23:50,233::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70233::DEBUG::2014-09-25 17:23:50,235::task::595::Storage.TaskManager.Task::(_updateState) Task=`9b2cb82f-23c2-4511-821a-74eea839bab4`::moving from state init -> state preparing >Thread-70233::INFO::2014-09-25 17:23:50,235::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70233::DEBUG::2014-09-25 17:23:50,236::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c4d118e8-a7df-4bfb-8ae0-89757045b859`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70233::DEBUG::2014-09-25 17:23:50,236::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70233::DEBUG::2014-09-25 17:23:50,236::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70233::DEBUG::2014-09-25 17:23:50,237::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c4d118e8-a7df-4bfb-8ae0-89757045b859`::Granted request >Thread-70233::DEBUG::2014-09-25 17:23:50,237::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9b2cb82f-23c2-4511-821a-74eea839bab4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70233::DEBUG::2014-09-25 17:23:50,237::task::993::Storage.TaskManager.Task::(_decref) Task=`9b2cb82f-23c2-4511-821a-74eea839bab4`::ref 1 aborting False >Thread-70233::INFO::2014-09-25 17:23:50,242::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70233::DEBUG::2014-09-25 17:23:50,242::task::1191::Storage.TaskManager.Task::(prepare) Task=`9b2cb82f-23c2-4511-821a-74eea839bab4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70233::DEBUG::2014-09-25 17:23:50,242::task::595::Storage.TaskManager.Task::(_updateState) Task=`9b2cb82f-23c2-4511-821a-74eea839bab4`::moving from state preparing -> state finished >Thread-70233::DEBUG::2014-09-25 17:23:50,242::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70233::DEBUG::2014-09-25 17:23:50,243::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70233::DEBUG::2014-09-25 17:23:50,243::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70233::DEBUG::2014-09-25 17:23:50,243::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70233::DEBUG::2014-09-25 17:23:50,244::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70233::DEBUG::2014-09-25 17:23:50,244::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70233::DEBUG::2014-09-25 17:23:50,244::task::993::Storage.TaskManager.Task::(_decref) Task=`9b2cb82f-23c2-4511-821a-74eea839bab4`::ref 0 aborting False >Thread-70233::DEBUG::2014-09-25 17:23:50,245::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70233::DEBUG::2014-09-25 17:23:50,245::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:51,362::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:51,364::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70234::DEBUG::2014-09-25 17:23:51,365::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:52,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:52,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218273 s, 46.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:53,639::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:53,641::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:53,642::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:53,644::__init__::232::IOProcess::(_processLogs) (108046) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:53,646::__init__::232::IOProcess::(_processLogs) (108046) Queuing response >Thread-80::DEBUG::2014-09-25 17:23:53,646::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:23:53,659::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000459084 s, 823 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:53,663::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:53,665::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:53,666::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:53,668::__init__::232::IOProcess::(_processLogs) (108047) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:53,670::__init__::232::IOProcess::(_processLogs) (108047) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:53,893::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:53,893::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:53,893::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:53,894::__init__::232::IOProcess::(_processLogs) (108048) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:53,899::__init__::232::IOProcess::(_processLogs) (108048) Queuing response >Thread-79::DEBUG::2014-09-25 17:23:53,900::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:23:53,919::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00251616 s, 223 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:53,920::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:53,921::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:53,921::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:53,921::__init__::232::IOProcess::(_processLogs) (108049) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:53,925::__init__::232::IOProcess::(_processLogs) (108049) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:53,926::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:53,927::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:53,927::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:53,927::__init__::232::IOProcess::(_processLogs) (108050) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:53,928::__init__::232::IOProcess::(_processLogs) (108050) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:53,929::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:53,929::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:53,930::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:53,931::__init__::232::IOProcess::(_processLogs) (108051) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:53,931::__init__::232::IOProcess::(_processLogs) (108051) Queuing response >Thread-29::DEBUG::2014-09-25 17:23:53,932::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:53,932::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:53,933::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:53,933::__init__::232::IOProcess::(_processLogs) (108052) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:23:53,934::__init__::232::IOProcess::(_processLogs) (108052) Queuing response >Dummy-148::DEBUG::2014-09-25 17:23:54,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:54,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194562 s, 52.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:54,378::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:54,379::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70235::DEBUG::2014-09-25 17:23:54,382::task::595::Storage.TaskManager.Task::(_updateState) Task=`d850c2db-90a8-4659-b9b4-2ff5baa7cc26`::moving from state init -> state preparing >Thread-70235::INFO::2014-09-25 17:23:54,382::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70235::INFO::2014-09-25 17:23:54,383::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00251616', 'lastCheck': '0.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000459084', 'lastCheck': '0.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000403655', 'lastCheck': '5.2', 'valid': True}} >Thread-70235::DEBUG::2014-09-25 17:23:54,383::task::1191::Storage.TaskManager.Task::(prepare) Task=`d850c2db-90a8-4659-b9b4-2ff5baa7cc26`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00251616', 'lastCheck': '0.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000459084', 'lastCheck': '0.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000403655', 'lastCheck': '5.2', 'valid': True}} >Thread-70235::DEBUG::2014-09-25 17:23:54,383::task::595::Storage.TaskManager.Task::(_updateState) Task=`d850c2db-90a8-4659-b9b4-2ff5baa7cc26`::moving from state preparing -> state finished >Thread-70235::DEBUG::2014-09-25 17:23:54,384::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70235::DEBUG::2014-09-25 17:23:54,384::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70235::DEBUG::2014-09-25 17:23:54,384::task::993::Storage.TaskManager.Task::(_decref) Task=`d850c2db-90a8-4659-b9b4-2ff5baa7cc26`::ref 0 aborting False >Thread-70235::DEBUG::2014-09-25 17:23:54,386::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:54,402::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:54,403::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70236::DEBUG::2014-09-25 17:23:54,405::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:56,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:56,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182593 s, 56.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:23:57,451::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:23:57,452::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70237::DEBUG::2014-09-25 17:23:57,454::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:23:58,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:23:58,233::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212194 s, 48.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:59,156::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:59,156::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:59,157::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:59,157::__init__::232::IOProcess::(_processLogs) (108053) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:59,172::__init__::232::IOProcess::(_processLogs) (108053) Queuing response >Thread-81::DEBUG::2014-09-25 17:23:59,173::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:23:59,199::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00339328 s, 109 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:23:59,200::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:23:59,201::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:23:59,201::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:23:59,201::__init__::232::IOProcess::(_processLogs) (108054) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:23:59,205::__init__::232::IOProcess::(_processLogs) (108054) Queuing response >Dummy-148::DEBUG::2014-09-25 17:24:00,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:00,269::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:00,271::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70238::DEBUG::2014-09-25 17:24:00,271::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70238::DEBUG::2014-09-25 17:24:00,273::task::595::Storage.TaskManager.Task::(_updateState) Task=`54da7331-84e5-43bc-9fac-94f12d6eea8b`::moving from state init -> state preparing >Thread-70238::INFO::2014-09-25 17:24:00,275::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70238::INFO::2014-09-25 17:24:00,275::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70238::DEBUG::2014-09-25 17:24:00,276::task::1191::Storage.TaskManager.Task::(prepare) Task=`54da7331-84e5-43bc-9fac-94f12d6eea8b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70238::DEBUG::2014-09-25 17:24:00,276::task::595::Storage.TaskManager.Task::(_updateState) Task=`54da7331-84e5-43bc-9fac-94f12d6eea8b`::moving from state preparing -> state finished >Thread-70238::DEBUG::2014-09-25 17:24:00,276::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70238::DEBUG::2014-09-25 17:24:00,276::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70238::DEBUG::2014-09-25 17:24:00,277::task::993::Storage.TaskManager.Task::(_decref) Task=`54da7331-84e5-43bc-9fac-94f12d6eea8b`::ref 0 aborting False >Thread-70238::DEBUG::2014-09-25 17:24:00,277::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70238::DEBUG::2014-09-25 17:24:00,277::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:00,291::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Dummy-148::DEBUG::2014-09-25 17:24:00,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0224255 s, 45.7 MB/s\n'; <rc> = 0 >JsonRpcServer::DEBUG::2014-09-25 17:24:00,295::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70239::DEBUG::2014-09-25 17:24:00,295::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70239::DEBUG::2014-09-25 17:24:00,299::task::595::Storage.TaskManager.Task::(_updateState) Task=`82700d01-497d-4ab1-ad56-94a2f88b45cb`::moving from state init -> state preparing >Thread-70239::INFO::2014-09-25 17:24:00,300::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70239::DEBUG::2014-09-25 17:24:00,302::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6cf365c3-5059-4c75-8aef-399f62579db0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70239::DEBUG::2014-09-25 17:24:00,303::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70239::DEBUG::2014-09-25 17:24:00,304::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70239::DEBUG::2014-09-25 17:24:00,305::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6cf365c3-5059-4c75-8aef-399f62579db0`::Granted request >Thread-70239::DEBUG::2014-09-25 17:24:00,307::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`82700d01-497d-4ab1-ad56-94a2f88b45cb`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70239::DEBUG::2014-09-25 17:24:00,308::task::993::Storage.TaskManager.Task::(_decref) Task=`82700d01-497d-4ab1-ad56-94a2f88b45cb`::ref 1 aborting False >Thread-70239::INFO::2014-09-25 17:24:00,323::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70239::DEBUG::2014-09-25 17:24:00,323::task::1191::Storage.TaskManager.Task::(prepare) Task=`82700d01-497d-4ab1-ad56-94a2f88b45cb`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70239::DEBUG::2014-09-25 17:24:00,324::task::595::Storage.TaskManager.Task::(_updateState) Task=`82700d01-497d-4ab1-ad56-94a2f88b45cb`::moving from state preparing -> state finished >Thread-70239::DEBUG::2014-09-25 17:24:00,325::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70239::DEBUG::2014-09-25 17:24:00,326::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70239::DEBUG::2014-09-25 17:24:00,327::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70239::DEBUG::2014-09-25 17:24:00,328::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70239::DEBUG::2014-09-25 17:24:00,329::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70239::DEBUG::2014-09-25 17:24:00,330::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70239::DEBUG::2014-09-25 17:24:00,331::task::993::Storage.TaskManager.Task::(_decref) Task=`82700d01-497d-4ab1-ad56-94a2f88b45cb`::ref 0 aborting False >Thread-70239::DEBUG::2014-09-25 17:24:00,333::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70239::DEBUG::2014-09-25 17:24:00,334::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:00,467::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:00,468::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70240::DEBUG::2014-09-25 17:24:00,470::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:02,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:02,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214165 s, 47.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:03,483::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:03,485::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70241::DEBUG::2014-09-25 17:24:03,487::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:24:03,674::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:03,674::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:03,674::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:03,674::__init__::232::IOProcess::(_processLogs) (108055) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:03,675::__init__::232::IOProcess::(_processLogs) (108055) Queuing response >Thread-80::DEBUG::2014-09-25 17:24:03,676::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:24:03,685::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00040088 s, 943 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:03,687::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:03,687::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:03,688::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:03,689::__init__::232::IOProcess::(_processLogs) (108056) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:03,689::__init__::232::IOProcess::(_processLogs) (108056) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:03,935::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:03,936::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:03,936::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:03,936::__init__::232::IOProcess::(_processLogs) (108057) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:03,936::__init__::232::IOProcess::(_processLogs) (108057) Queuing response >Thread-79::DEBUG::2014-09-25 17:24:03,938::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:24:03,961::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00690433 s, 81.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:03,962::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:03,963::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:03,963::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:03,963::__init__::232::IOProcess::(_processLogs) (108058) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:03,971::__init__::232::IOProcess::(_processLogs) (108058) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:03,973::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:03,973::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:03,974::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:03,974::__init__::232::IOProcess::(_processLogs) (108059) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:03,974::__init__::232::IOProcess::(_processLogs) (108059) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:03,975::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:03,975::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:03,976::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:03,976::__init__::232::IOProcess::(_processLogs) (108060) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:03,976::__init__::232::IOProcess::(_processLogs) (108060) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:03,977::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:03,977::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:03,977::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:03,978::__init__::232::IOProcess::(_processLogs) (108061) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:03,978::__init__::232::IOProcess::(_processLogs) (108061) Queuing response >Dummy-148::DEBUG::2014-09-25 17:24:04,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:04,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0230699 s, 44.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:24:06,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:06,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155539 s, 65.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:06,505::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:06,509::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70242::DEBUG::2014-09-25 17:24:06,513::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:08,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:08,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199896 s, 51.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:09,207::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:09,208::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:09,208::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:09,208::__init__::232::IOProcess::(_processLogs) (108062) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:09,213::__init__::232::IOProcess::(_processLogs) (108062) Queuing response >Thread-81::DEBUG::2014-09-25 17:24:09,214::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:24:09,227::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00248692 s, 149 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:09,228::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:09,229::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:09,229::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:09,229::__init__::232::IOProcess::(_processLogs) (108063) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:09,232::__init__::232::IOProcess::(_processLogs) (108063) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:09,531::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:09,533::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70243::DEBUG::2014-09-25 17:24:09,535::task::595::Storage.TaskManager.Task::(_updateState) Task=`5f84d91e-cf50-4fcf-a969-d180f5769753`::moving from state init -> state preparing >Thread-70243::INFO::2014-09-25 17:24:09,536::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70243::INFO::2014-09-25 17:24:09,536::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00690433', 'lastCheck': '5.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00040088', 'lastCheck': '5.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00248692', 'lastCheck': '0.3', 'valid': True}} >Thread-70243::DEBUG::2014-09-25 17:24:09,537::task::1191::Storage.TaskManager.Task::(prepare) Task=`5f84d91e-cf50-4fcf-a969-d180f5769753`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00690433', 'lastCheck': '5.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00040088', 'lastCheck': '5.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00248692', 'lastCheck': '0.3', 'valid': True}} >Thread-70243::DEBUG::2014-09-25 17:24:09,537::task::595::Storage.TaskManager.Task::(_updateState) Task=`5f84d91e-cf50-4fcf-a969-d180f5769753`::moving from state preparing -> state finished >Thread-70243::DEBUG::2014-09-25 17:24:09,537::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70243::DEBUG::2014-09-25 17:24:09,537::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70243::DEBUG::2014-09-25 17:24:09,537::task::993::Storage.TaskManager.Task::(_decref) Task=`5f84d91e-cf50-4fcf-a969-d180f5769753`::ref 0 aborting False >Thread-70243::DEBUG::2014-09-25 17:24:09,540::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:09,563::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:09,565::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70244::DEBUG::2014-09-25 17:24:09,567::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:10,360::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70245::DEBUG::2014-09-25 17:24:10,361::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:24:10,362::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70245::DEBUG::2014-09-25 17:24:10,364::task::595::Storage.TaskManager.Task::(_updateState) Task=`1ee04eed-7f58-4847-92e6-c985db6af778`::moving from state init -> state preparing >Thread-70245::INFO::2014-09-25 17:24:10,364::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70245::INFO::2014-09-25 17:24:10,365::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70245::DEBUG::2014-09-25 17:24:10,365::task::1191::Storage.TaskManager.Task::(prepare) Task=`1ee04eed-7f58-4847-92e6-c985db6af778`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70245::DEBUG::2014-09-25 17:24:10,365::task::595::Storage.TaskManager.Task::(_updateState) Task=`1ee04eed-7f58-4847-92e6-c985db6af778`::moving from state preparing -> state finished >Thread-70245::DEBUG::2014-09-25 17:24:10,365::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70245::DEBUG::2014-09-25 17:24:10,366::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70245::DEBUG::2014-09-25 17:24:10,366::task::993::Storage.TaskManager.Task::(_decref) Task=`1ee04eed-7f58-4847-92e6-c985db6af778`::ref 0 aborting False >Thread-70245::DEBUG::2014-09-25 17:24:10,366::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70245::DEBUG::2014-09-25 17:24:10,366::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:10,380::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70246::DEBUG::2014-09-25 17:24:10,381::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:24:10,381::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70246::DEBUG::2014-09-25 17:24:10,383::task::595::Storage.TaskManager.Task::(_updateState) Task=`d5c10101-81b5-4b1f-91d4-c2ae3c792427`::moving from state init -> state preparing >Thread-70246::INFO::2014-09-25 17:24:10,383::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70246::DEBUG::2014-09-25 17:24:10,384::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5f07f341-4268-4853-a1cb-cf53c9bcd1c9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70246::DEBUG::2014-09-25 17:24:10,384::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70246::DEBUG::2014-09-25 17:24:10,385::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70246::DEBUG::2014-09-25 17:24:10,385::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5f07f341-4268-4853-a1cb-cf53c9bcd1c9`::Granted request >Thread-70246::DEBUG::2014-09-25 17:24:10,386::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`d5c10101-81b5-4b1f-91d4-c2ae3c792427`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70246::DEBUG::2014-09-25 17:24:10,386::task::993::Storage.TaskManager.Task::(_decref) Task=`d5c10101-81b5-4b1f-91d4-c2ae3c792427`::ref 1 aborting False >Thread-70246::INFO::2014-09-25 17:24:10,390::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70246::DEBUG::2014-09-25 17:24:10,391::task::1191::Storage.TaskManager.Task::(prepare) Task=`d5c10101-81b5-4b1f-91d4-c2ae3c792427`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70246::DEBUG::2014-09-25 17:24:10,391::task::595::Storage.TaskManager.Task::(_updateState) Task=`d5c10101-81b5-4b1f-91d4-c2ae3c792427`::moving from state preparing -> state finished >Thread-70246::DEBUG::2014-09-25 17:24:10,391::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70246::DEBUG::2014-09-25 17:24:10,391::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70246::DEBUG::2014-09-25 17:24:10,392::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70246::DEBUG::2014-09-25 17:24:10,392::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70246::DEBUG::2014-09-25 17:24:10,392::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70246::DEBUG::2014-09-25 17:24:10,392::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70246::DEBUG::2014-09-25 17:24:10,393::task::993::Storage.TaskManager.Task::(_decref) Task=`d5c10101-81b5-4b1f-91d4-c2ae3c792427`::ref 0 aborting False >Thread-70246::DEBUG::2014-09-25 17:24:10,393::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70246::DEBUG::2014-09-25 17:24:10,394::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:10,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:10,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152454 s, 67.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:12,618::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:12,619::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70247::DEBUG::2014-09-25 17:24:12,622::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:12,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:12,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134586 s, 76.1 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 17:24:13,689::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:24:13,693::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:13,694::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:13,695::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:13,695::__init__::232::IOProcess::(_processLogs) (108064) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:24:13,695::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:13,695::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:13,696::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:13,696::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:13,696::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:13,696::__init__::232::IOProcess::(_processLogs) (108065) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:24:13,697::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:13,697::__init__::232::IOProcess::(_processLogs) (108066) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:24:13,698::__init__::232::IOProcess::(_processLogs) (108065) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:13,700::__init__::232::IOProcess::(_processLogs) (108064) Queuing response >Thread-80::DEBUG::2014-09-25 17:24:13,704::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:24:13,705::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:13,705::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:13,705::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:13,705::__init__::232::IOProcess::(_processLogs) (108067) Got request for method 'touch' >Thread-80::DEBUG::2014-09-25 17:24:13,709::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:24:13,709::__init__::232::IOProcess::(_processLogs) (108067) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:13,711::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:13,711::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:13,712::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:13,712::__init__::232::IOProcess::(_processLogs) (108068) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:24:13,712::__init__::232::IOProcess::(_processLogs) (108068) Queuing response >Thread-80::DEBUG::2014-09-25 17:24:13,713::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:24:13,714::__init__::232::IOProcess::(_processLogs) (108066) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:13,717::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:13,717::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:13,718::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:13,718::__init__::232::IOProcess::(_processLogs) (108069) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:13,718::__init__::232::IOProcess::(_processLogs) (108069) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:13,719::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:13,719::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:13,719::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:13,720::__init__::232::IOProcess::(_processLogs) (108070) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:24:13,720::__init__::232::IOProcess::(_processLogs) (108070) Queuing response >Thread-80::DEBUG::2014-09-25 17:24:13,720::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-80::INFO::2014-09-25 17:24:13,721::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:24:13,722::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:24:13,722::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:13,723::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:13,723::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:13,724::__init__::232::IOProcess::(_processLogs) (108071) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:24:13,724::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:24:13,725::__init__::232::IOProcess::(_processLogs) (108071) Queuing response >Thread-80::DEBUG::2014-09-25 17:24:13,734::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00034567 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:13,736::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:13,736::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:13,736::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:13,736::__init__::232::IOProcess::(_processLogs) (108072) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:13,737::__init__::232::IOProcess::(_processLogs) (108072) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:13,979::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:13,979::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:13,980::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:13,980::__init__::232::IOProcess::(_processLogs) (108073) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:13,996::__init__::232::IOProcess::(_processLogs) (108073) Queuing response >Thread-79::DEBUG::2014-09-25 17:24:13,997::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:24:14,010::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0038105 s, 147 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:14,011::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:14,011::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:14,011::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:14,012::__init__::232::IOProcess::(_processLogs) (108074) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:14,020::__init__::232::IOProcess::(_processLogs) (108074) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:14,021::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:14,022::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:14,022::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:14,023::__init__::232::IOProcess::(_processLogs) (108075) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:14,023::__init__::232::IOProcess::(_processLogs) (108075) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:14,023::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:14,024::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:14,025::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:14,025::__init__::232::IOProcess::(_processLogs) (108076) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:14,025::__init__::232::IOProcess::(_processLogs) (108076) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:14,026::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:14,026::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:14,027::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:14,027::__init__::232::IOProcess::(_processLogs) (108077) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:14,028::__init__::232::IOProcess::(_processLogs) (108077) Queuing response >Dummy-148::DEBUG::2014-09-25 17:24:14,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:14,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199179 s, 51.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:15,635::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:15,637::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70251::DEBUG::2014-09-25 17:24:15,639::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:16,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:16,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0249562 s, 41.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:18,650::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:18,652::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70252::DEBUG::2014-09-25 17:24:18,654::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:18,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:18,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131812 s, 77.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:19,234::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:19,234::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:19,234::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:19,235::__init__::232::IOProcess::(_processLogs) (108078) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:19,237::__init__::232::IOProcess::(_processLogs) (108078) Queuing response >Thread-81::DEBUG::2014-09-25 17:24:19,239::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:24:19,265::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00314613 s, 118 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:19,266::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:19,267::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:19,267::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:19,267::__init__::232::IOProcess::(_processLogs) (108079) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:19,277::__init__::232::IOProcess::(_processLogs) (108079) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:20,430::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70253::DEBUG::2014-09-25 17:24:20,431::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:24:20,431::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70253::DEBUG::2014-09-25 17:24:20,434::task::595::Storage.TaskManager.Task::(_updateState) Task=`1411e39b-d255-40c9-b1ab-1b2da858d51e`::moving from state init -> state preparing >Thread-70253::INFO::2014-09-25 17:24:20,434::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70253::INFO::2014-09-25 17:24:20,435::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70253::DEBUG::2014-09-25 17:24:20,435::task::1191::Storage.TaskManager.Task::(prepare) Task=`1411e39b-d255-40c9-b1ab-1b2da858d51e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70253::DEBUG::2014-09-25 17:24:20,435::task::595::Storage.TaskManager.Task::(_updateState) Task=`1411e39b-d255-40c9-b1ab-1b2da858d51e`::moving from state preparing -> state finished >Thread-70253::DEBUG::2014-09-25 17:24:20,435::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70253::DEBUG::2014-09-25 17:24:20,436::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70253::DEBUG::2014-09-25 17:24:20,436::task::993::Storage.TaskManager.Task::(_decref) Task=`1411e39b-d255-40c9-b1ab-1b2da858d51e`::ref 0 aborting False >Thread-70253::DEBUG::2014-09-25 17:24:20,436::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70253::DEBUG::2014-09-25 17:24:20,437::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:20,452::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:20,453::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70254::DEBUG::2014-09-25 17:24:20,453::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70254::DEBUG::2014-09-25 17:24:20,455::task::595::Storage.TaskManager.Task::(_updateState) Task=`4dfa2196-22a3-4808-8b9c-21c5100dc5d0`::moving from state init -> state preparing >Thread-70254::INFO::2014-09-25 17:24:20,455::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70254::DEBUG::2014-09-25 17:24:20,456::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c46a33a0-dd0c-4f63-acdb-704751c432c3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70254::DEBUG::2014-09-25 17:24:20,456::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70254::DEBUG::2014-09-25 17:24:20,456::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70254::DEBUG::2014-09-25 17:24:20,457::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c46a33a0-dd0c-4f63-acdb-704751c432c3`::Granted request >Thread-70254::DEBUG::2014-09-25 17:24:20,457::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4dfa2196-22a3-4808-8b9c-21c5100dc5d0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70254::DEBUG::2014-09-25 17:24:20,457::task::993::Storage.TaskManager.Task::(_decref) Task=`4dfa2196-22a3-4808-8b9c-21c5100dc5d0`::ref 1 aborting False >Thread-70254::INFO::2014-09-25 17:24:20,462::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70254::DEBUG::2014-09-25 17:24:20,462::task::1191::Storage.TaskManager.Task::(prepare) Task=`4dfa2196-22a3-4808-8b9c-21c5100dc5d0`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70254::DEBUG::2014-09-25 17:24:20,462::task::595::Storage.TaskManager.Task::(_updateState) Task=`4dfa2196-22a3-4808-8b9c-21c5100dc5d0`::moving from state preparing -> state finished >Thread-70254::DEBUG::2014-09-25 17:24:20,463::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70254::DEBUG::2014-09-25 17:24:20,463::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70254::DEBUG::2014-09-25 17:24:20,463::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70254::DEBUG::2014-09-25 17:24:20,463::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70254::DEBUG::2014-09-25 17:24:20,464::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70254::DEBUG::2014-09-25 17:24:20,464::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70254::DEBUG::2014-09-25 17:24:20,464::task::993::Storage.TaskManager.Task::(_decref) Task=`4dfa2196-22a3-4808-8b9c-21c5100dc5d0`::ref 0 aborting False >Thread-70254::DEBUG::2014-09-25 17:24:20,465::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70254::DEBUG::2014-09-25 17:24:20,465::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:21,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:21,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019783 s, 51.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:21,673::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:21,674::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70255::DEBUG::2014-09-25 17:24:21,676::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:23,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:23,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171511 s, 59.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:23,739::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:23,740::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:23,740::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:23,740::__init__::232::IOProcess::(_processLogs) (108080) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:23,741::__init__::232::IOProcess::(_processLogs) (108080) Queuing response >Thread-80::DEBUG::2014-09-25 17:24:23,742::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:24:23,750::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000374539 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:23,752::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:23,752::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:23,752::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:23,753::__init__::232::IOProcess::(_processLogs) (108081) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:23,753::__init__::232::IOProcess::(_processLogs) (108081) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:24,028::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:24,028::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:24,028::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:24,030::__init__::232::IOProcess::(_processLogs) (108082) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 17:24:24,030::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:24:24,031::__init__::232::IOProcess::(_processLogs) (108082) Queuing response >Thread-79::DEBUG::2014-09-25 17:24:24,040::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000365926 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:24,041::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:24,041::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:24,041::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:24,043::__init__::232::IOProcess::(_processLogs) (108083) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:24,043::__init__::232::IOProcess::(_processLogs) (108083) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:24,044::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:24,044::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:24,044::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:24,045::__init__::232::IOProcess::(_processLogs) (108084) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:24,045::__init__::232::IOProcess::(_processLogs) (108084) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:24,046::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:24,046::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:24,047::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:24,047::__init__::232::IOProcess::(_processLogs) (108085) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:24,047::__init__::232::IOProcess::(_processLogs) (108085) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:24,048::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:24,048::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:24,049::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:24,050::__init__::232::IOProcess::(_processLogs) (108086) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:24,050::__init__::232::IOProcess::(_processLogs) (108086) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:24,695::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:24,696::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70256::DEBUG::2014-09-25 17:24:24,698::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d0e0ac3-85c0-477a-a7bd-cbcb73069190`::moving from state init -> state preparing >Thread-70256::INFO::2014-09-25 17:24:24,699::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70256::INFO::2014-09-25 17:24:24,699::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000365926', 'lastCheck': '0.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000374539', 'lastCheck': '0.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00314613', 'lastCheck': '5.4', 'valid': True}} >Thread-70256::DEBUG::2014-09-25 17:24:24,699::task::1191::Storage.TaskManager.Task::(prepare) Task=`6d0e0ac3-85c0-477a-a7bd-cbcb73069190`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000365926', 'lastCheck': '0.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000374539', 'lastCheck': '0.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00314613', 'lastCheck': '5.4', 'valid': True}} >Thread-70256::DEBUG::2014-09-25 17:24:24,700::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d0e0ac3-85c0-477a-a7bd-cbcb73069190`::moving from state preparing -> state finished >Thread-70256::DEBUG::2014-09-25 17:24:24,700::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70256::DEBUG::2014-09-25 17:24:24,700::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70256::DEBUG::2014-09-25 17:24:24,700::task::993::Storage.TaskManager.Task::(_decref) Task=`6d0e0ac3-85c0-477a-a7bd-cbcb73069190`::ref 0 aborting False >Thread-70256::DEBUG::2014-09-25 17:24:24,703::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:24,726::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:24,727::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70257::DEBUG::2014-09-25 17:24:24,729::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:25,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:25,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143565 s, 71.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:24:27,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:27,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131993 s, 77.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:27,787::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:27,788::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70258::DEBUG::2014-09-25 17:24:27,790::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:29,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:29,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213388 s, 48.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:29,278::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:29,279::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:29,279::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:29,279::__init__::232::IOProcess::(_processLogs) (108087) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:29,285::__init__::232::IOProcess::(_processLogs) (108087) Queuing response >Thread-81::DEBUG::2014-09-25 17:24:29,286::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:24:29,301::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000773993 s, 479 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:29,302::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:29,302::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:29,302::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:29,303::__init__::232::IOProcess::(_processLogs) (108088) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:29,304::__init__::232::IOProcess::(_processLogs) (108088) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:30,496::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70259::DEBUG::2014-09-25 17:24:30,497::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:24:30,497::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70259::DEBUG::2014-09-25 17:24:30,499::task::595::Storage.TaskManager.Task::(_updateState) Task=`e2c6f263-148f-4325-af6d-507c711c83b9`::moving from state init -> state preparing >Thread-70259::INFO::2014-09-25 17:24:30,500::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70259::INFO::2014-09-25 17:24:30,500::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70259::DEBUG::2014-09-25 17:24:30,500::task::1191::Storage.TaskManager.Task::(prepare) Task=`e2c6f263-148f-4325-af6d-507c711c83b9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70259::DEBUG::2014-09-25 17:24:30,501::task::595::Storage.TaskManager.Task::(_updateState) Task=`e2c6f263-148f-4325-af6d-507c711c83b9`::moving from state preparing -> state finished >Thread-70259::DEBUG::2014-09-25 17:24:30,501::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70259::DEBUG::2014-09-25 17:24:30,501::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70259::DEBUG::2014-09-25 17:24:30,501::task::993::Storage.TaskManager.Task::(_decref) Task=`e2c6f263-148f-4325-af6d-507c711c83b9`::ref 0 aborting False >Thread-70259::DEBUG::2014-09-25 17:24:30,502::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70259::DEBUG::2014-09-25 17:24:30,502::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:30,513::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70260::DEBUG::2014-09-25 17:24:30,514::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:24:30,514::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70260::DEBUG::2014-09-25 17:24:30,517::task::595::Storage.TaskManager.Task::(_updateState) Task=`ce409611-3531-4e3a-ac38-52bbdb89a2d6`::moving from state init -> state preparing >Thread-70260::INFO::2014-09-25 17:24:30,517::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70260::DEBUG::2014-09-25 17:24:30,518::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c459534d-85b3-4cdf-80c0-849457ab3eb3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70260::DEBUG::2014-09-25 17:24:30,518::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70260::DEBUG::2014-09-25 17:24:30,518::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70260::DEBUG::2014-09-25 17:24:30,518::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c459534d-85b3-4cdf-80c0-849457ab3eb3`::Granted request >Thread-70260::DEBUG::2014-09-25 17:24:30,519::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ce409611-3531-4e3a-ac38-52bbdb89a2d6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70260::DEBUG::2014-09-25 17:24:30,519::task::993::Storage.TaskManager.Task::(_decref) Task=`ce409611-3531-4e3a-ac38-52bbdb89a2d6`::ref 1 aborting False >Thread-70260::INFO::2014-09-25 17:24:30,524::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70260::DEBUG::2014-09-25 17:24:30,524::task::1191::Storage.TaskManager.Task::(prepare) Task=`ce409611-3531-4e3a-ac38-52bbdb89a2d6`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70260::DEBUG::2014-09-25 17:24:30,524::task::595::Storage.TaskManager.Task::(_updateState) Task=`ce409611-3531-4e3a-ac38-52bbdb89a2d6`::moving from state preparing -> state finished >Thread-70260::DEBUG::2014-09-25 17:24:30,524::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70260::DEBUG::2014-09-25 17:24:30,525::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70260::DEBUG::2014-09-25 17:24:30,525::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70260::DEBUG::2014-09-25 17:24:30,525::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70260::DEBUG::2014-09-25 17:24:30,525::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70260::DEBUG::2014-09-25 17:24:30,526::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70260::DEBUG::2014-09-25 17:24:30,526::task::993::Storage.TaskManager.Task::(_decref) Task=`ce409611-3531-4e3a-ac38-52bbdb89a2d6`::ref 0 aborting False >Thread-70260::DEBUG::2014-09-25 17:24:30,526::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70260::DEBUG::2014-09-25 17:24:30,527::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:30,804::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:30,805::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70261::DEBUG::2014-09-25 17:24:30,807::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:31,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:31,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019972 s, 51.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:24:33,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:33,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153732 s, 66.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:33,755::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:33,756::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:33,756::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:33,756::__init__::232::IOProcess::(_processLogs) (108089) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:33,757::__init__::232::IOProcess::(_processLogs) (108089) Queuing response >Thread-80::DEBUG::2014-09-25 17:24:33,758::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:24:33,767::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000517209 s, 731 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:33,768::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:33,768::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:33,768::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:33,769::__init__::232::IOProcess::(_processLogs) (108090) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:33,769::__init__::232::IOProcess::(_processLogs) (108090) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:33,824::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:33,826::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70262::DEBUG::2014-09-25 17:24:33,827::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:24:34,050::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:34,051::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:34,051::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:34,051::__init__::232::IOProcess::(_processLogs) (108091) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:34,053::__init__::232::IOProcess::(_processLogs) (108091) Queuing response >Thread-79::DEBUG::2014-09-25 17:24:34,053::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:24:34,062::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000429248 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:34,063::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:34,063::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:34,064::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:34,064::__init__::232::IOProcess::(_processLogs) (108092) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:34,065::__init__::232::IOProcess::(_processLogs) (108092) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:34,066::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:34,067::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:34,067::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:34,068::__init__::232::IOProcess::(_processLogs) (108093) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:34,068::__init__::232::IOProcess::(_processLogs) (108093) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:34,068::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:34,068::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:34,069::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:34,070::__init__::232::IOProcess::(_processLogs) (108094) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:34,070::__init__::232::IOProcess::(_processLogs) (108094) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:34,070::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:34,071::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:34,072::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:34,072::__init__::232::IOProcess::(_processLogs) (108095) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:34,072::__init__::232::IOProcess::(_processLogs) (108095) Queuing response >Dummy-148::DEBUG::2014-09-25 17:24:35,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:35,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018755 s, 54.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:36,842::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:36,843::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70263::DEBUG::2014-09-25 17:24:36,845::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:37,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:37,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0270479 s, 37.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:39,306::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:39,307::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:39,307::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:39,307::__init__::232::IOProcess::(_processLogs) (108096) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:39,310::__init__::232::IOProcess::(_processLogs) (108096) Queuing response >Thread-81::DEBUG::2014-09-25 17:24:39,311::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:24:39,325::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00270756 s, 137 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:39,327::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:39,327::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:39,327::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:39,327::__init__::232::IOProcess::(_processLogs) (108097) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:39,330::__init__::232::IOProcess::(_processLogs) (108097) Queuing response >Dummy-148::DEBUG::2014-09-25 17:24:39,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:39,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168019 s, 60.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:39,861::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:39,862::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70264::DEBUG::2014-09-25 17:24:39,865::task::595::Storage.TaskManager.Task::(_updateState) Task=`17211804-653d-43f1-bd63-f9af0db415f2`::moving from state init -> state preparing >Thread-70264::INFO::2014-09-25 17:24:39,865::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70264::INFO::2014-09-25 17:24:39,865::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000429248', 'lastCheck': '5.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000517209', 'lastCheck': '6.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00270756', 'lastCheck': '0.5', 'valid': True}} >Thread-70264::DEBUG::2014-09-25 17:24:39,866::task::1191::Storage.TaskManager.Task::(prepare) Task=`17211804-653d-43f1-bd63-f9af0db415f2`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000429248', 'lastCheck': '5.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000517209', 'lastCheck': '6.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00270756', 'lastCheck': '0.5', 'valid': True}} >Thread-70264::DEBUG::2014-09-25 17:24:39,866::task::595::Storage.TaskManager.Task::(_updateState) Task=`17211804-653d-43f1-bd63-f9af0db415f2`::moving from state preparing -> state finished >Thread-70264::DEBUG::2014-09-25 17:24:39,866::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70264::DEBUG::2014-09-25 17:24:39,866::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70264::DEBUG::2014-09-25 17:24:39,867::task::993::Storage.TaskManager.Task::(_decref) Task=`17211804-653d-43f1-bd63-f9af0db415f2`::ref 0 aborting False >Thread-70264::DEBUG::2014-09-25 17:24:39,869::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:39,889::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:39,890::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70265::DEBUG::2014-09-25 17:24:39,892::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:40,554::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70266::DEBUG::2014-09-25 17:24:40,555::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:24:40,555::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70266::DEBUG::2014-09-25 17:24:40,557::task::595::Storage.TaskManager.Task::(_updateState) Task=`c80f9c97-c710-449f-8faf-525efcf0aaff`::moving from state init -> state preparing >Thread-70266::INFO::2014-09-25 17:24:40,558::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70266::INFO::2014-09-25 17:24:40,558::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70266::DEBUG::2014-09-25 17:24:40,558::task::1191::Storage.TaskManager.Task::(prepare) Task=`c80f9c97-c710-449f-8faf-525efcf0aaff`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70266::DEBUG::2014-09-25 17:24:40,559::task::595::Storage.TaskManager.Task::(_updateState) Task=`c80f9c97-c710-449f-8faf-525efcf0aaff`::moving from state preparing -> state finished >Thread-70266::DEBUG::2014-09-25 17:24:40,559::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70266::DEBUG::2014-09-25 17:24:40,559::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70266::DEBUG::2014-09-25 17:24:40,559::task::993::Storage.TaskManager.Task::(_decref) Task=`c80f9c97-c710-449f-8faf-525efcf0aaff`::ref 0 aborting False >Thread-70266::DEBUG::2014-09-25 17:24:40,560::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70266::DEBUG::2014-09-25 17:24:40,560::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:40,574::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:40,576::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70267::DEBUG::2014-09-25 17:24:40,576::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70267::DEBUG::2014-09-25 17:24:40,577::task::595::Storage.TaskManager.Task::(_updateState) Task=`abd76150-1fe5-4ad9-b248-b13ebd8a34db`::moving from state init -> state preparing >Thread-70267::INFO::2014-09-25 17:24:40,578::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70267::DEBUG::2014-09-25 17:24:40,578::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a787b43b-e314-41d7-a2a0-60bbbcd1cf07`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70267::DEBUG::2014-09-25 17:24:40,579::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70267::DEBUG::2014-09-25 17:24:40,580::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70267::DEBUG::2014-09-25 17:24:40,580::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a787b43b-e314-41d7-a2a0-60bbbcd1cf07`::Granted request >Thread-70267::DEBUG::2014-09-25 17:24:40,581::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`abd76150-1fe5-4ad9-b248-b13ebd8a34db`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70267::DEBUG::2014-09-25 17:24:40,581::task::993::Storage.TaskManager.Task::(_decref) Task=`abd76150-1fe5-4ad9-b248-b13ebd8a34db`::ref 1 aborting False >Thread-70267::INFO::2014-09-25 17:24:40,585::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70267::DEBUG::2014-09-25 17:24:40,586::task::1191::Storage.TaskManager.Task::(prepare) Task=`abd76150-1fe5-4ad9-b248-b13ebd8a34db`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70267::DEBUG::2014-09-25 17:24:40,586::task::595::Storage.TaskManager.Task::(_updateState) Task=`abd76150-1fe5-4ad9-b248-b13ebd8a34db`::moving from state preparing -> state finished >Thread-70267::DEBUG::2014-09-25 17:24:40,586::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70267::DEBUG::2014-09-25 17:24:40,586::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70267::DEBUG::2014-09-25 17:24:40,587::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70267::DEBUG::2014-09-25 17:24:40,587::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70267::DEBUG::2014-09-25 17:24:40,587::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70267::DEBUG::2014-09-25 17:24:40,588::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70267::DEBUG::2014-09-25 17:24:40,588::task::993::Storage.TaskManager.Task::(_decref) Task=`abd76150-1fe5-4ad9-b248-b13ebd8a34db`::ref 0 aborting False >Thread-70267::DEBUG::2014-09-25 17:24:40,588::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70267::DEBUG::2014-09-25 17:24:40,589::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:41,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:41,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132258 s, 77.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:42,956::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:42,958::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70268::DEBUG::2014-09-25 17:24:42,960::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:43,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:43,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177771 s, 57.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:43,771::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:43,772::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:43,772::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:43,772::__init__::232::IOProcess::(_processLogs) (108098) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:43,773::__init__::232::IOProcess::(_processLogs) (108098) Queuing response >Thread-80::DEBUG::2014-09-25 17:24:43,774::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:24:43,783::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000486479 s, 777 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:43,785::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:43,785::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:43,785::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:43,786::__init__::232::IOProcess::(_processLogs) (108099) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:43,786::__init__::232::IOProcess::(_processLogs) (108099) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:44,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:44,073::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:44,073::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:24:44,074::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:24:44,075::__init__::232::IOProcess::(_processLogs) (108100) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:44,083::__init__::232::IOProcess::(_processLogs) (108100) Queuing response >Thread-79::DEBUG::2014-09-25 17:24:44,084::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000916999 s, 613 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:44,085::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:44,086::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:44,086::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:44,086::__init__::232::IOProcess::(_processLogs) (108101) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:44,087::__init__::232::IOProcess::(_processLogs) (108101) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:44,088::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:44,088::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:44,089::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:44,089::__init__::232::IOProcess::(_processLogs) (108102) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:44,089::__init__::232::IOProcess::(_processLogs) (108102) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:44,090::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:44,090::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:44,091::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:44,091::__init__::232::IOProcess::(_processLogs) (108103) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:44,091::__init__::232::IOProcess::(_processLogs) (108103) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:44,092::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:44,092::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:44,093::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:44,093::__init__::232::IOProcess::(_processLogs) (108104) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:44,094::__init__::232::IOProcess::(_processLogs) (108104) Queuing response >Dummy-148::DEBUG::2014-09-25 17:24:45,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:45,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144269 s, 71.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:45,976::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:45,978::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70269::DEBUG::2014-09-25 17:24:45,980::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:47,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:47,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157429 s, 65.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:48,995::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:48,996::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70270::DEBUG::2014-09-25 17:24:48,998::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:24:49,332::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:49,332::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:49,333::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:49,333::__init__::232::IOProcess::(_processLogs) (108105) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:49,342::__init__::232::IOProcess::(_processLogs) (108105) Queuing response >Thread-81::DEBUG::2014-09-25 17:24:49,343::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:24:49,361::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00624027 s, 59.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:49,362::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:49,362::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:49,362::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:49,363::__init__::232::IOProcess::(_processLogs) (108106) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:49,364::__init__::232::IOProcess::(_processLogs) (108106) Queuing response >Dummy-148::DEBUG::2014-09-25 17:24:49,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:49,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159076 s, 64.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:50,617::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:50,619::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70271::DEBUG::2014-09-25 17:24:50,619::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70271::DEBUG::2014-09-25 17:24:50,621::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6ac3b42-c48f-4613-89b7-29c94de447db`::moving from state init -> state preparing >Thread-70271::INFO::2014-09-25 17:24:50,621::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70271::INFO::2014-09-25 17:24:50,622::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70271::DEBUG::2014-09-25 17:24:50,622::task::1191::Storage.TaskManager.Task::(prepare) Task=`b6ac3b42-c48f-4613-89b7-29c94de447db`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70271::DEBUG::2014-09-25 17:24:50,622::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6ac3b42-c48f-4613-89b7-29c94de447db`::moving from state preparing -> state finished >Thread-70271::DEBUG::2014-09-25 17:24:50,622::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70271::DEBUG::2014-09-25 17:24:50,623::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70271::DEBUG::2014-09-25 17:24:50,623::task::993::Storage.TaskManager.Task::(_decref) Task=`b6ac3b42-c48f-4613-89b7-29c94de447db`::ref 0 aborting False >Thread-70271::DEBUG::2014-09-25 17:24:50,623::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70271::DEBUG::2014-09-25 17:24:50,624::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:50,638::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:50,639::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70272::DEBUG::2014-09-25 17:24:50,639::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70272::DEBUG::2014-09-25 17:24:50,641::task::595::Storage.TaskManager.Task::(_updateState) Task=`6bbb6a0d-57dd-46de-a125-e4a3c4a45df0`::moving from state init -> state preparing >Thread-70272::INFO::2014-09-25 17:24:50,641::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70272::DEBUG::2014-09-25 17:24:50,641::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`defa2354-6f73-4525-b811-e0320e378047`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70272::DEBUG::2014-09-25 17:24:50,642::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70272::DEBUG::2014-09-25 17:24:50,642::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70272::DEBUG::2014-09-25 17:24:50,642::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`defa2354-6f73-4525-b811-e0320e378047`::Granted request >Thread-70272::DEBUG::2014-09-25 17:24:50,643::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`6bbb6a0d-57dd-46de-a125-e4a3c4a45df0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70272::DEBUG::2014-09-25 17:24:50,643::task::993::Storage.TaskManager.Task::(_decref) Task=`6bbb6a0d-57dd-46de-a125-e4a3c4a45df0`::ref 1 aborting False >Thread-70272::INFO::2014-09-25 17:24:50,648::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70272::DEBUG::2014-09-25 17:24:50,648::task::1191::Storage.TaskManager.Task::(prepare) Task=`6bbb6a0d-57dd-46de-a125-e4a3c4a45df0`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70272::DEBUG::2014-09-25 17:24:50,648::task::595::Storage.TaskManager.Task::(_updateState) Task=`6bbb6a0d-57dd-46de-a125-e4a3c4a45df0`::moving from state preparing -> state finished >Thread-70272::DEBUG::2014-09-25 17:24:50,648::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70272::DEBUG::2014-09-25 17:24:50,649::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70272::DEBUG::2014-09-25 17:24:50,649::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70272::DEBUG::2014-09-25 17:24:50,649::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70272::DEBUG::2014-09-25 17:24:50,649::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70272::DEBUG::2014-09-25 17:24:50,650::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70272::DEBUG::2014-09-25 17:24:50,650::task::993::Storage.TaskManager.Task::(_decref) Task=`6bbb6a0d-57dd-46de-a125-e4a3c4a45df0`::ref 0 aborting False >Thread-70272::DEBUG::2014-09-25 17:24:50,650::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70272::DEBUG::2014-09-25 17:24:50,651::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:51,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:51,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152292 s, 67.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:52,012::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:52,014::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70273::DEBUG::2014-09-25 17:24:52,016::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:24:53,788::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:53,789::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:53,789::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:53,789::__init__::232::IOProcess::(_processLogs) (108107) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:53,790::__init__::232::IOProcess::(_processLogs) (108107) Queuing response >Thread-80::DEBUG::2014-09-25 17:24:53,791::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:53,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-80::DEBUG::2014-09-25 17:24:53,808::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000511081 s, 740 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:53,809::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:53,809::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:53,810::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:53,812::__init__::232::IOProcess::(_processLogs) (108108) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:53,812::__init__::232::IOProcess::(_processLogs) (108108) Queuing response >Dummy-148::DEBUG::2014-09-25 17:24:53,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131228 s, 78.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:54,095::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:54,095::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:54,095::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:54,096::__init__::232::IOProcess::(_processLogs) (108109) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 17:24:54,097::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:24:54,097::__init__::232::IOProcess::(_processLogs) (108109) Queuing response >Thread-79::DEBUG::2014-09-25 17:24:54,107::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000481572 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:54,108::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:54,108::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:54,108::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:54,109::__init__::232::IOProcess::(_processLogs) (108110) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:54,109::__init__::232::IOProcess::(_processLogs) (108110) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:54,110::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:54,111::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:54,111::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:54,112::__init__::232::IOProcess::(_processLogs) (108111) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:54,112::__init__::232::IOProcess::(_processLogs) (108111) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:54,112::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:54,113::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:54,113::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:54,114::__init__::232::IOProcess::(_processLogs) (108112) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:54,114::__init__::232::IOProcess::(_processLogs) (108112) Queuing response >Thread-29::DEBUG::2014-09-25 17:24:54,115::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:54,115::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:54,115::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:54,116::__init__::232::IOProcess::(_processLogs) (108113) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:24:54,116::__init__::232::IOProcess::(_processLogs) (108113) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:55,026::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:55,027::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70274::DEBUG::2014-09-25 17:24:55,030::task::595::Storage.TaskManager.Task::(_updateState) Task=`a6920fa3-bd4b-46d6-aa46-abbc825870d7`::moving from state init -> state preparing >Thread-70274::INFO::2014-09-25 17:24:55,030::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70274::INFO::2014-09-25 17:24:55,030::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000481572', 'lastCheck': '0.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000511081', 'lastCheck': '1.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00624027', 'lastCheck': '5.7', 'valid': True}} >Thread-70274::DEBUG::2014-09-25 17:24:55,031::task::1191::Storage.TaskManager.Task::(prepare) Task=`a6920fa3-bd4b-46d6-aa46-abbc825870d7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000481572', 'lastCheck': '0.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000511081', 'lastCheck': '1.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00624027', 'lastCheck': '5.7', 'valid': True}} >Thread-70274::DEBUG::2014-09-25 17:24:55,031::task::595::Storage.TaskManager.Task::(_updateState) Task=`a6920fa3-bd4b-46d6-aa46-abbc825870d7`::moving from state preparing -> state finished >Thread-70274::DEBUG::2014-09-25 17:24:55,031::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70274::DEBUG::2014-09-25 17:24:55,031::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70274::DEBUG::2014-09-25 17:24:55,032::task::993::Storage.TaskManager.Task::(_decref) Task=`a6920fa3-bd4b-46d6-aa46-abbc825870d7`::ref 0 aborting False >Thread-70274::DEBUG::2014-09-25 17:24:55,034::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:55,050::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:55,051::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70275::DEBUG::2014-09-25 17:24:55,053::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:24:55,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:55,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205886 s, 49.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:24:57,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:57,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136249 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:24:58,102::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:24:58,104::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70276::DEBUG::2014-09-25 17:24:58,106::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:24:59,367::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:59,367::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:59,367::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:59,367::__init__::232::IOProcess::(_processLogs) (108114) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:59,368::__init__::232::IOProcess::(_processLogs) (108114) Queuing response >Thread-81::DEBUG::2014-09-25 17:24:59,369::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:24:59,378::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00039202 s, 946 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:24:59,379::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:24:59,379::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:24:59,379::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:24:59,380::__init__::232::IOProcess::(_processLogs) (108115) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:24:59,380::__init__::232::IOProcess::(_processLogs) (108115) Queuing response >Dummy-148::DEBUG::2014-09-25 17:24:59,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:24:59,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016218 s, 63.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:00,677::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:00,678::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70277::DEBUG::2014-09-25 17:25:00,679::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70277::DEBUG::2014-09-25 17:25:00,680::task::595::Storage.TaskManager.Task::(_updateState) Task=`de3c3fa0-1586-41a5-bfd0-a5b8a9f29fab`::moving from state init -> state preparing >Thread-70277::INFO::2014-09-25 17:25:00,681::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70277::INFO::2014-09-25 17:25:00,681::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70277::DEBUG::2014-09-25 17:25:00,682::task::1191::Storage.TaskManager.Task::(prepare) Task=`de3c3fa0-1586-41a5-bfd0-a5b8a9f29fab`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70277::DEBUG::2014-09-25 17:25:00,682::task::595::Storage.TaskManager.Task::(_updateState) Task=`de3c3fa0-1586-41a5-bfd0-a5b8a9f29fab`::moving from state preparing -> state finished >Thread-70277::DEBUG::2014-09-25 17:25:00,682::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70277::DEBUG::2014-09-25 17:25:00,682::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70277::DEBUG::2014-09-25 17:25:00,683::task::993::Storage.TaskManager.Task::(_decref) Task=`de3c3fa0-1586-41a5-bfd0-a5b8a9f29fab`::ref 0 aborting False >Thread-70277::DEBUG::2014-09-25 17:25:00,683::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70277::DEBUG::2014-09-25 17:25:00,683::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:00,694::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:00,695::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70278::DEBUG::2014-09-25 17:25:00,696::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70278::DEBUG::2014-09-25 17:25:00,697::task::595::Storage.TaskManager.Task::(_updateState) Task=`86877d97-efaa-4c44-9e8d-55b7a0c211e8`::moving from state init -> state preparing >Thread-70278::INFO::2014-09-25 17:25:00,697::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70278::DEBUG::2014-09-25 17:25:00,698::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`14ccb8ae-1817-42c2-8fd5-e4a160c2dfc9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70278::DEBUG::2014-09-25 17:25:00,698::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70278::DEBUG::2014-09-25 17:25:00,699::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70278::DEBUG::2014-09-25 17:25:00,699::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`14ccb8ae-1817-42c2-8fd5-e4a160c2dfc9`::Granted request >Thread-70278::DEBUG::2014-09-25 17:25:00,699::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`86877d97-efaa-4c44-9e8d-55b7a0c211e8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70278::DEBUG::2014-09-25 17:25:00,700::task::993::Storage.TaskManager.Task::(_decref) Task=`86877d97-efaa-4c44-9e8d-55b7a0c211e8`::ref 1 aborting False >Thread-70278::INFO::2014-09-25 17:25:00,704::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70278::DEBUG::2014-09-25 17:25:00,704::task::1191::Storage.TaskManager.Task::(prepare) Task=`86877d97-efaa-4c44-9e8d-55b7a0c211e8`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70278::DEBUG::2014-09-25 17:25:00,705::task::595::Storage.TaskManager.Task::(_updateState) Task=`86877d97-efaa-4c44-9e8d-55b7a0c211e8`::moving from state preparing -> state finished >Thread-70278::DEBUG::2014-09-25 17:25:00,705::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70278::DEBUG::2014-09-25 17:25:00,705::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70278::DEBUG::2014-09-25 17:25:00,705::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70278::DEBUG::2014-09-25 17:25:00,706::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70278::DEBUG::2014-09-25 17:25:00,706::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70278::DEBUG::2014-09-25 17:25:00,706::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70278::DEBUG::2014-09-25 17:25:00,707::task::993::Storage.TaskManager.Task::(_decref) Task=`86877d97-efaa-4c44-9e8d-55b7a0c211e8`::ref 0 aborting False >Thread-70278::DEBUG::2014-09-25 17:25:00,707::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70278::DEBUG::2014-09-25 17:25:00,708::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:01,119::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:01,120::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70279::DEBUG::2014-09-25 17:25:01,122::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:01,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:02,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020859 s, 49.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:03,812::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:03,813::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:03,813::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:03,813::__init__::232::IOProcess::(_processLogs) (108116) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:03,814::__init__::232::IOProcess::(_processLogs) (108116) Queuing response >Thread-80::DEBUG::2014-09-25 17:25:03,815::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:25:03,824::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000486309 s, 777 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:03,825::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:03,825::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:03,826::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:03,826::__init__::232::IOProcess::(_processLogs) (108117) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:03,827::__init__::232::IOProcess::(_processLogs) (108117) Queuing response >Dummy-148::DEBUG::2014-09-25 17:25:04,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:04,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0240381 s, 42.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:04,119::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:04,120::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:04,122::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:04,123::__init__::232::IOProcess::(_processLogs) (108118) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:04,131::__init__::232::IOProcess::(_processLogs) (108118) Queuing response >Thread-79::DEBUG::2014-09-25 17:25:04,134::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:04,135::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:04,149::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70280::DEBUG::2014-09-25 17:25:04,158::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-79::DEBUG::2014-09-25 17:25:04,164::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00698829 s, 80.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:04,168::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:04,170::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:04,171::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:04,173::__init__::232::IOProcess::(_processLogs) (108119) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:04,177::__init__::232::IOProcess::(_processLogs) (108119) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:04,181::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:04,183::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:04,185::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:04,187::__init__::232::IOProcess::(_processLogs) (108120) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:04,189::__init__::232::IOProcess::(_processLogs) (108120) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:04,192::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:04,194::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:04,196::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:04,198::__init__::232::IOProcess::(_processLogs) (108121) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:04,199::__init__::232::IOProcess::(_processLogs) (108121) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:04,203::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:04,205::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:04,207::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:04,209::__init__::232::IOProcess::(_processLogs) (108122) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:04,211::__init__::232::IOProcess::(_processLogs) (108122) Queuing response >Dummy-148::DEBUG::2014-09-25 17:25:06,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:06,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0283904 s, 36.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:07,175::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:07,176::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70281::DEBUG::2014-09-25 17:25:07,178::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:08,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:08,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133466 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:09,382::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:09,383::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:09,383::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:09,383::__init__::232::IOProcess::(_processLogs) (108123) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:09,384::__init__::232::IOProcess::(_processLogs) (108123) Queuing response >Thread-81::DEBUG::2014-09-25 17:25:09,385::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:25:09,394::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000382094 s, 971 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:09,395::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:09,395::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:09,395::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:09,396::__init__::232::IOProcess::(_processLogs) (108124) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:09,408::__init__::232::IOProcess::(_processLogs) (108124) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:10,189::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:10,190::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70282::DEBUG::2014-09-25 17:25:10,193::task::595::Storage.TaskManager.Task::(_updateState) Task=`abc0eb41-0021-4111-a85e-38a53f7c3089`::moving from state init -> state preparing >Thread-70282::INFO::2014-09-25 17:25:10,193::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70282::INFO::2014-09-25 17:25:10,193::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00698829', 'lastCheck': '6.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000486309', 'lastCheck': '6.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000382094', 'lastCheck': '0.8', 'valid': True}} >Thread-70282::DEBUG::2014-09-25 17:25:10,194::task::1191::Storage.TaskManager.Task::(prepare) Task=`abc0eb41-0021-4111-a85e-38a53f7c3089`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00698829', 'lastCheck': '6.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000486309', 'lastCheck': '6.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000382094', 'lastCheck': '0.8', 'valid': True}} >Thread-70282::DEBUG::2014-09-25 17:25:10,194::task::595::Storage.TaskManager.Task::(_updateState) Task=`abc0eb41-0021-4111-a85e-38a53f7c3089`::moving from state preparing -> state finished >Thread-70282::DEBUG::2014-09-25 17:25:10,194::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70282::DEBUG::2014-09-25 17:25:10,195::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70282::DEBUG::2014-09-25 17:25:10,195::task::993::Storage.TaskManager.Task::(_decref) Task=`abc0eb41-0021-4111-a85e-38a53f7c3089`::ref 0 aborting False >Thread-70282::DEBUG::2014-09-25 17:25:10,197::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:10,210::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:10,211::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70283::DEBUG::2014-09-25 17:25:10,213::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:10,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:10,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160693 s, 63.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:10,733::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70284::DEBUG::2014-09-25 17:25:10,734::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:25:10,734::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70284::DEBUG::2014-09-25 17:25:10,737::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c2726da-d7d2-4e01-8fa2-08996946421b`::moving from state init -> state preparing >Thread-70284::INFO::2014-09-25 17:25:10,737::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70284::INFO::2014-09-25 17:25:10,737::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70284::DEBUG::2014-09-25 17:25:10,738::task::1191::Storage.TaskManager.Task::(prepare) Task=`3c2726da-d7d2-4e01-8fa2-08996946421b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70284::DEBUG::2014-09-25 17:25:10,738::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c2726da-d7d2-4e01-8fa2-08996946421b`::moving from state preparing -> state finished >Thread-70284::DEBUG::2014-09-25 17:25:10,738::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70284::DEBUG::2014-09-25 17:25:10,738::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70284::DEBUG::2014-09-25 17:25:10,739::task::993::Storage.TaskManager.Task::(_decref) Task=`3c2726da-d7d2-4e01-8fa2-08996946421b`::ref 0 aborting False >Thread-70284::DEBUG::2014-09-25 17:25:10,739::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70284::DEBUG::2014-09-25 17:25:10,739::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:10,754::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:10,756::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70285::DEBUG::2014-09-25 17:25:10,756::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70285::DEBUG::2014-09-25 17:25:10,757::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a8abe3c-bc30-4059-a7bd-0b2679d428df`::moving from state init -> state preparing >Thread-70285::INFO::2014-09-25 17:25:10,758::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70285::DEBUG::2014-09-25 17:25:10,758::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`50cbbc6b-9903-44f7-a126-7502418acd8c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70285::DEBUG::2014-09-25 17:25:10,759::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70285::DEBUG::2014-09-25 17:25:10,759::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70285::DEBUG::2014-09-25 17:25:10,759::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`50cbbc6b-9903-44f7-a126-7502418acd8c`::Granted request >Thread-70285::DEBUG::2014-09-25 17:25:10,760::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1a8abe3c-bc30-4059-a7bd-0b2679d428df`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70285::DEBUG::2014-09-25 17:25:10,760::task::993::Storage.TaskManager.Task::(_decref) Task=`1a8abe3c-bc30-4059-a7bd-0b2679d428df`::ref 1 aborting False >Thread-70285::INFO::2014-09-25 17:25:10,765::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70285::DEBUG::2014-09-25 17:25:10,765::task::1191::Storage.TaskManager.Task::(prepare) Task=`1a8abe3c-bc30-4059-a7bd-0b2679d428df`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70285::DEBUG::2014-09-25 17:25:10,765::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a8abe3c-bc30-4059-a7bd-0b2679d428df`::moving from state preparing -> state finished >Thread-70285::DEBUG::2014-09-25 17:25:10,765::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70285::DEBUG::2014-09-25 17:25:10,766::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70285::DEBUG::2014-09-25 17:25:10,766::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70285::DEBUG::2014-09-25 17:25:10,766::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70285::DEBUG::2014-09-25 17:25:10,767::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70285::DEBUG::2014-09-25 17:25:10,767::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70285::DEBUG::2014-09-25 17:25:10,767::task::993::Storage.TaskManager.Task::(_decref) Task=`1a8abe3c-bc30-4059-a7bd-0b2679d428df`::ref 0 aborting False >Thread-70285::DEBUG::2014-09-25 17:25:10,768::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70285::DEBUG::2014-09-25 17:25:10,768::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:12,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:12,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146581 s, 69.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:13,256::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:13,258::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70286::DEBUG::2014-09-25 17:25:13,260::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:25:13,829::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:13,829::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:13,829::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:13,829::__init__::232::IOProcess::(_processLogs) (108125) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:13,830::__init__::232::IOProcess::(_processLogs) (108125) Queuing response >Thread-80::DEBUG::2014-09-25 17:25:13,831::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:25:13,840::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000577183 s, 655 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:13,841::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:13,842::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:13,842::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:13,842::__init__::232::IOProcess::(_processLogs) (108126) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:13,843::__init__::232::IOProcess::(_processLogs) (108126) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:14,214::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:14,215::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:14,215::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:14,215::__init__::232::IOProcess::(_processLogs) (108127) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:14,219::__init__::232::IOProcess::(_processLogs) (108127) Queuing response >Thread-79::DEBUG::2014-09-25 17:25:14,220::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:25:14,237::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0041466 s, 136 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:14,238::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:14,238::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:14,239::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:14,239::__init__::232::IOProcess::(_processLogs) (108128) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:14,244::__init__::232::IOProcess::(_processLogs) (108128) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:14,245::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:14,245::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:14,245::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:14,246::__init__::232::IOProcess::(_processLogs) (108129) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:14,246::__init__::232::IOProcess::(_processLogs) (108129) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:14,247::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:14,247::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:14,248::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:14,248::__init__::232::IOProcess::(_processLogs) (108130) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:14,249::__init__::232::IOProcess::(_processLogs) (108130) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:14,249::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:14,250::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:14,251::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:14,251::__init__::232::IOProcess::(_processLogs) (108131) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:14,251::__init__::232::IOProcess::(_processLogs) (108131) Queuing response >Dummy-148::DEBUG::2014-09-25 17:25:14,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:14,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0197615 s, 51.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:16,273::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:16,276::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70287::DEBUG::2014-09-25 17:25:16,281::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:16,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:16,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0327163 s, 31.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:25:18,774::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:18,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164683 s, 62.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:19,297::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:19,298::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70288::DEBUG::2014-09-25 17:25:19,300::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:25:19,409::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:19,410::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:19,410::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:19,410::__init__::232::IOProcess::(_processLogs) (108132) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:19,416::__init__::232::IOProcess::(_processLogs) (108132) Queuing response >Thread-81::DEBUG::2014-09-25 17:25:19,417::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:25:19,426::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000325985 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:19,427::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:19,427::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:19,428::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:19,428::__init__::232::IOProcess::(_processLogs) (108133) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:19,428::__init__::232::IOProcess::(_processLogs) (108133) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:20,796::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:20,798::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70289::DEBUG::2014-09-25 17:25:20,798::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70289::DEBUG::2014-09-25 17:25:20,800::task::595::Storage.TaskManager.Task::(_updateState) Task=`91094bd8-657f-47b9-81d2-1a3a07650627`::moving from state init -> state preparing >Thread-70289::INFO::2014-09-25 17:25:20,800::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70289::INFO::2014-09-25 17:25:20,800::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70289::DEBUG::2014-09-25 17:25:20,801::task::1191::Storage.TaskManager.Task::(prepare) Task=`91094bd8-657f-47b9-81d2-1a3a07650627`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70289::DEBUG::2014-09-25 17:25:20,801::task::595::Storage.TaskManager.Task::(_updateState) Task=`91094bd8-657f-47b9-81d2-1a3a07650627`::moving from state preparing -> state finished >Thread-70289::DEBUG::2014-09-25 17:25:20,801::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70289::DEBUG::2014-09-25 17:25:20,802::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70289::DEBUG::2014-09-25 17:25:20,802::task::993::Storage.TaskManager.Task::(_decref) Task=`91094bd8-657f-47b9-81d2-1a3a07650627`::ref 0 aborting False >Thread-70289::DEBUG::2014-09-25 17:25:20,803::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70289::DEBUG::2014-09-25 17:25:20,803::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:20,813::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:20,814::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70290::DEBUG::2014-09-25 17:25:20,814::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70290::DEBUG::2014-09-25 17:25:20,816::task::595::Storage.TaskManager.Task::(_updateState) Task=`f55d384b-41a5-44f4-9efa-15b1f27fffc0`::moving from state init -> state preparing >Thread-70290::INFO::2014-09-25 17:25:20,816::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70290::DEBUG::2014-09-25 17:25:20,817::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b55e1bf5-25e9-47e1-beb0-458ec8924710`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70290::DEBUG::2014-09-25 17:25:20,817::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70290::DEBUG::2014-09-25 17:25:20,818::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70290::DEBUG::2014-09-25 17:25:20,818::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b55e1bf5-25e9-47e1-beb0-458ec8924710`::Granted request >Thread-70290::DEBUG::2014-09-25 17:25:20,818::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f55d384b-41a5-44f4-9efa-15b1f27fffc0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70290::DEBUG::2014-09-25 17:25:20,819::task::993::Storage.TaskManager.Task::(_decref) Task=`f55d384b-41a5-44f4-9efa-15b1f27fffc0`::ref 1 aborting False >Dummy-148::DEBUG::2014-09-25 17:25:20,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-70290::INFO::2014-09-25 17:25:20,833::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70290::DEBUG::2014-09-25 17:25:20,834::task::1191::Storage.TaskManager.Task::(prepare) Task=`f55d384b-41a5-44f4-9efa-15b1f27fffc0`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70290::DEBUG::2014-09-25 17:25:20,834::task::595::Storage.TaskManager.Task::(_updateState) Task=`f55d384b-41a5-44f4-9efa-15b1f27fffc0`::moving from state preparing -> state finished >Thread-70290::DEBUG::2014-09-25 17:25:20,834::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70290::DEBUG::2014-09-25 17:25:20,834::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70290::DEBUG::2014-09-25 17:25:20,835::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70290::DEBUG::2014-09-25 17:25:20,835::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70290::DEBUG::2014-09-25 17:25:20,836::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70290::DEBUG::2014-09-25 17:25:20,836::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70290::DEBUG::2014-09-25 17:25:20,836::task::993::Storage.TaskManager.Task::(_decref) Task=`f55d384b-41a5-44f4-9efa-15b1f27fffc0`::ref 0 aborting False >Thread-70290::DEBUG::2014-09-25 17:25:20,837::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70290::DEBUG::2014-09-25 17:25:20,837::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:20,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169371 s, 60.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:22,316::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:22,318::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70291::DEBUG::2014-09-25 17:25:22,320::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:22,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:22,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0206746 s, 49.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:23,845::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:23,846::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:23,846::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:23,846::__init__::232::IOProcess::(_processLogs) (108134) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:23,846::__init__::232::IOProcess::(_processLogs) (108134) Queuing response >Thread-80::DEBUG::2014-09-25 17:25:23,848::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:25:23,857::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000618861 s, 611 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:23,858::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:23,858::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:23,859::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:23,859::__init__::232::IOProcess::(_processLogs) (108135) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:23,860::__init__::232::IOProcess::(_processLogs) (108135) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:24,251::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:24,252::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:24,252::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:24,252::__init__::232::IOProcess::(_processLogs) (108136) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:24,260::__init__::232::IOProcess::(_processLogs) (108136) Queuing response >Thread-79::DEBUG::2014-09-25 17:25:24,261::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:25:24,279::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00801264 s, 70.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:24,280::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:24,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:24,281::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:24,281::__init__::232::IOProcess::(_processLogs) (108137) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:24,287::__init__::232::IOProcess::(_processLogs) (108137) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:24,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:24,289::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:24,289::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:24,290::__init__::232::IOProcess::(_processLogs) (108138) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:24,290::__init__::232::IOProcess::(_processLogs) (108138) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:24,291::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:24,291::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:24,292::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:24,292::__init__::232::IOProcess::(_processLogs) (108139) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:24,292::__init__::232::IOProcess::(_processLogs) (108139) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:24,293::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:24,293::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:24,294::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:24,295::__init__::232::IOProcess::(_processLogs) (108140) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:24,295::__init__::232::IOProcess::(_processLogs) (108140) Queuing response >Dummy-148::DEBUG::2014-09-25 17:25:24,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:24,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153406 s, 66.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:25,332::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:25,334::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70292::DEBUG::2014-09-25 17:25:25,337::task::595::Storage.TaskManager.Task::(_updateState) Task=`42b55012-3c97-4cb0-ae4a-ba94f587f5f8`::moving from state init -> state preparing >Thread-70292::INFO::2014-09-25 17:25:25,337::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70292::INFO::2014-09-25 17:25:25,338::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00801264', 'lastCheck': '1.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000618861', 'lastCheck': '1.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000325985', 'lastCheck': '5.9', 'valid': True}} >Thread-70292::DEBUG::2014-09-25 17:25:25,338::task::1191::Storage.TaskManager.Task::(prepare) Task=`42b55012-3c97-4cb0-ae4a-ba94f587f5f8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00801264', 'lastCheck': '1.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000618861', 'lastCheck': '1.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000325985', 'lastCheck': '5.9', 'valid': True}} >Thread-70292::DEBUG::2014-09-25 17:25:25,338::task::595::Storage.TaskManager.Task::(_updateState) Task=`42b55012-3c97-4cb0-ae4a-ba94f587f5f8`::moving from state preparing -> state finished >Thread-70292::DEBUG::2014-09-25 17:25:25,339::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70292::DEBUG::2014-09-25 17:25:25,339::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70292::DEBUG::2014-09-25 17:25:25,339::task::993::Storage.TaskManager.Task::(_decref) Task=`42b55012-3c97-4cb0-ae4a-ba94f587f5f8`::ref 0 aborting False >Thread-70292::DEBUG::2014-09-25 17:25:25,341::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:25,356::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:25,356::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70293::DEBUG::2014-09-25 17:25:25,358::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:26,973::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:27,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019945 s, 51.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:28,408::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:28,409::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70294::DEBUG::2014-09-25 17:25:28,411::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:29,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:29,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013349 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:29,431::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:29,431::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:29,431::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:29,432::__init__::232::IOProcess::(_processLogs) (108141) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:29,436::__init__::232::IOProcess::(_processLogs) (108141) Queuing response >Thread-81::DEBUG::2014-09-25 17:25:29,438::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:25:29,452::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00449733 s, 82.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:29,453::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:29,453::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:29,453::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:29,454::__init__::232::IOProcess::(_processLogs) (108142) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:29,459::__init__::232::IOProcess::(_processLogs) (108142) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:30,866::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70295::DEBUG::2014-09-25 17:25:30,868::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:25:30,868::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70295::DEBUG::2014-09-25 17:25:30,870::task::595::Storage.TaskManager.Task::(_updateState) Task=`194dc6a2-5b59-456b-92d6-0998398f5584`::moving from state init -> state preparing >Thread-70295::INFO::2014-09-25 17:25:30,870::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70295::INFO::2014-09-25 17:25:30,871::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70295::DEBUG::2014-09-25 17:25:30,871::task::1191::Storage.TaskManager.Task::(prepare) Task=`194dc6a2-5b59-456b-92d6-0998398f5584`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70295::DEBUG::2014-09-25 17:25:30,871::task::595::Storage.TaskManager.Task::(_updateState) Task=`194dc6a2-5b59-456b-92d6-0998398f5584`::moving from state preparing -> state finished >Thread-70295::DEBUG::2014-09-25 17:25:30,872::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70295::DEBUG::2014-09-25 17:25:30,872::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70295::DEBUG::2014-09-25 17:25:30,872::task::993::Storage.TaskManager.Task::(_decref) Task=`194dc6a2-5b59-456b-92d6-0998398f5584`::ref 0 aborting False >Thread-70295::DEBUG::2014-09-25 17:25:30,872::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70295::DEBUG::2014-09-25 17:25:30,873::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:30,889::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70296::DEBUG::2014-09-25 17:25:30,890::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:25:30,890::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70296::DEBUG::2014-09-25 17:25:30,892::task::595::Storage.TaskManager.Task::(_updateState) Task=`9eb39746-33f1-48f5-adba-56a3b3097d80`::moving from state init -> state preparing >Thread-70296::INFO::2014-09-25 17:25:30,892::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70296::DEBUG::2014-09-25 17:25:30,893::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`abd02c92-7f9c-4421-8ed9-f20729f95038`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70296::DEBUG::2014-09-25 17:25:30,893::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70296::DEBUG::2014-09-25 17:25:30,893::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70296::DEBUG::2014-09-25 17:25:30,894::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`abd02c92-7f9c-4421-8ed9-f20729f95038`::Granted request >Thread-70296::DEBUG::2014-09-25 17:25:30,894::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9eb39746-33f1-48f5-adba-56a3b3097d80`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70296::DEBUG::2014-09-25 17:25:30,895::task::993::Storage.TaskManager.Task::(_decref) Task=`9eb39746-33f1-48f5-adba-56a3b3097d80`::ref 1 aborting False >Thread-70296::INFO::2014-09-25 17:25:30,899::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70296::DEBUG::2014-09-25 17:25:30,899::task::1191::Storage.TaskManager.Task::(prepare) Task=`9eb39746-33f1-48f5-adba-56a3b3097d80`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70296::DEBUG::2014-09-25 17:25:30,899::task::595::Storage.TaskManager.Task::(_updateState) Task=`9eb39746-33f1-48f5-adba-56a3b3097d80`::moving from state preparing -> state finished >Thread-70296::DEBUG::2014-09-25 17:25:30,900::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70296::DEBUG::2014-09-25 17:25:30,900::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70296::DEBUG::2014-09-25 17:25:30,900::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70296::DEBUG::2014-09-25 17:25:30,900::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70296::DEBUG::2014-09-25 17:25:30,901::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70296::DEBUG::2014-09-25 17:25:30,901::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70296::DEBUG::2014-09-25 17:25:30,901::task::993::Storage.TaskManager.Task::(_decref) Task=`9eb39746-33f1-48f5-adba-56a3b3097d80`::ref 0 aborting False >Thread-70296::DEBUG::2014-09-25 17:25:30,902::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70296::DEBUG::2014-09-25 17:25:30,902::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:31,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:31,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165832 s, 61.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:31,428::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:31,429::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70297::DEBUG::2014-09-25 17:25:31,431::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:33,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:33,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157087 s, 65.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:33,862::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:33,862::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:33,862::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:33,862::__init__::232::IOProcess::(_processLogs) (108143) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:33,863::__init__::232::IOProcess::(_processLogs) (108143) Queuing response >Thread-80::DEBUG::2014-09-25 17:25:33,864::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:25:33,873::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000478407 s, 790 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:33,874::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:33,875::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:33,875::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:33,875::__init__::232::IOProcess::(_processLogs) (108144) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:33,876::__init__::232::IOProcess::(_processLogs) (108144) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:34,295::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:34,295::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:34,296::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:34,296::__init__::232::IOProcess::(_processLogs) (108145) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:34,302::__init__::232::IOProcess::(_processLogs) (108145) Queuing response >Thread-79::DEBUG::2014-09-25 17:25:34,304::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:25:34,321::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00198659 s, 283 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:34,322::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:34,322::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:34,322::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:34,323::__init__::232::IOProcess::(_processLogs) (108146) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:34,328::__init__::232::IOProcess::(_processLogs) (108146) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:34,329::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:34,330::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:34,331::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:34,331::__init__::232::IOProcess::(_processLogs) (108147) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:34,331::__init__::232::IOProcess::(_processLogs) (108147) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:34,331::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:34,332::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:34,332::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:34,332::__init__::232::IOProcess::(_processLogs) (108148) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:34,332::__init__::232::IOProcess::(_processLogs) (108148) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:34,333::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:34,334::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:34,335::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:34,335::__init__::232::IOProcess::(_processLogs) (108149) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:34,335::__init__::232::IOProcess::(_processLogs) (108149) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:34,446::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:34,449::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70298::DEBUG::2014-09-25 17:25:34,455::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:35,163::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:35,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185438 s, 55.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:25:37,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:37,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188042 s, 54.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:37,471::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:37,473::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70299::DEBUG::2014-09-25 17:25:37,475::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:39,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:39,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015529 s, 65.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:39,461::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:39,461::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:39,462::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:39,462::__init__::232::IOProcess::(_processLogs) (108150) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:39,464::__init__::232::IOProcess::(_processLogs) (108150) Queuing response >Thread-81::DEBUG::2014-09-25 17:25:39,465::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:25:39,494::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00569852 s, 65.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:39,496::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:39,496::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:39,496::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:39,496::__init__::232::IOProcess::(_processLogs) (108151) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:39,503::__init__::232::IOProcess::(_processLogs) (108151) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:40,486::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:40,487::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70300::DEBUG::2014-09-25 17:25:40,490::task::595::Storage.TaskManager.Task::(_updateState) Task=`81cc5bf4-4fdd-4f1c-8204-c0a34f72e251`::moving from state init -> state preparing >Thread-70300::INFO::2014-09-25 17:25:40,491::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70300::INFO::2014-09-25 17:25:40,491::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00198659', 'lastCheck': '6.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000478407', 'lastCheck': '6.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00569852', 'lastCheck': '1.0', 'valid': True}} >Thread-70300::DEBUG::2014-09-25 17:25:40,491::task::1191::Storage.TaskManager.Task::(prepare) Task=`81cc5bf4-4fdd-4f1c-8204-c0a34f72e251`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00198659', 'lastCheck': '6.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000478407', 'lastCheck': '6.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00569852', 'lastCheck': '1.0', 'valid': True}} >Thread-70300::DEBUG::2014-09-25 17:25:40,492::task::595::Storage.TaskManager.Task::(_updateState) Task=`81cc5bf4-4fdd-4f1c-8204-c0a34f72e251`::moving from state preparing -> state finished >Thread-70300::DEBUG::2014-09-25 17:25:40,492::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70300::DEBUG::2014-09-25 17:25:40,492::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70300::DEBUG::2014-09-25 17:25:40,492::task::993::Storage.TaskManager.Task::(_decref) Task=`81cc5bf4-4fdd-4f1c-8204-c0a34f72e251`::ref 0 aborting False >Thread-70300::DEBUG::2014-09-25 17:25:40,495::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:40,509::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:40,510::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70301::DEBUG::2014-09-25 17:25:40,512::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:40,930::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:40,931::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70302::DEBUG::2014-09-25 17:25:40,931::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70302::DEBUG::2014-09-25 17:25:40,932::task::595::Storage.TaskManager.Task::(_updateState) Task=`f32320c5-f993-413d-810e-175f0eda7762`::moving from state init -> state preparing >Thread-70302::INFO::2014-09-25 17:25:40,933::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70302::INFO::2014-09-25 17:25:40,933::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70302::DEBUG::2014-09-25 17:25:40,934::task::1191::Storage.TaskManager.Task::(prepare) Task=`f32320c5-f993-413d-810e-175f0eda7762`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70302::DEBUG::2014-09-25 17:25:40,934::task::595::Storage.TaskManager.Task::(_updateState) Task=`f32320c5-f993-413d-810e-175f0eda7762`::moving from state preparing -> state finished >Thread-70302::DEBUG::2014-09-25 17:25:40,934::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70302::DEBUG::2014-09-25 17:25:40,934::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70302::DEBUG::2014-09-25 17:25:40,934::task::993::Storage.TaskManager.Task::(_decref) Task=`f32320c5-f993-413d-810e-175f0eda7762`::ref 0 aborting False >Thread-70302::DEBUG::2014-09-25 17:25:40,935::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70302::DEBUG::2014-09-25 17:25:40,935::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:40,946::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:40,947::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70303::DEBUG::2014-09-25 17:25:40,947::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70303::DEBUG::2014-09-25 17:25:40,949::task::595::Storage.TaskManager.Task::(_updateState) Task=`48ab0edd-64bd-4fe1-8fa4-924883b52fc0`::moving from state init -> state preparing >Thread-70303::INFO::2014-09-25 17:25:40,949::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70303::DEBUG::2014-09-25 17:25:40,950::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8f703939-f13a-4216-acb4-d0d6d4700f3d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70303::DEBUG::2014-09-25 17:25:40,950::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70303::DEBUG::2014-09-25 17:25:40,950::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70303::DEBUG::2014-09-25 17:25:40,951::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8f703939-f13a-4216-acb4-d0d6d4700f3d`::Granted request >Thread-70303::DEBUG::2014-09-25 17:25:40,951::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`48ab0edd-64bd-4fe1-8fa4-924883b52fc0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70303::DEBUG::2014-09-25 17:25:40,951::task::993::Storage.TaskManager.Task::(_decref) Task=`48ab0edd-64bd-4fe1-8fa4-924883b52fc0`::ref 1 aborting False >Thread-70303::INFO::2014-09-25 17:25:40,956::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70303::DEBUG::2014-09-25 17:25:40,956::task::1191::Storage.TaskManager.Task::(prepare) Task=`48ab0edd-64bd-4fe1-8fa4-924883b52fc0`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70303::DEBUG::2014-09-25 17:25:40,956::task::595::Storage.TaskManager.Task::(_updateState) Task=`48ab0edd-64bd-4fe1-8fa4-924883b52fc0`::moving from state preparing -> state finished >Thread-70303::DEBUG::2014-09-25 17:25:40,956::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70303::DEBUG::2014-09-25 17:25:40,957::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70303::DEBUG::2014-09-25 17:25:40,957::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70303::DEBUG::2014-09-25 17:25:40,957::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70303::DEBUG::2014-09-25 17:25:40,957::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70303::DEBUG::2014-09-25 17:25:40,958::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70303::DEBUG::2014-09-25 17:25:40,958::task::993::Storage.TaskManager.Task::(_decref) Task=`48ab0edd-64bd-4fe1-8fa4-924883b52fc0`::ref 0 aborting False >Thread-70303::DEBUG::2014-09-25 17:25:40,959::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70303::DEBUG::2014-09-25 17:25:40,959::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:41,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:41,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132757 s, 77.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:25:43,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:43,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181441 s, 56.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:43,557::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:43,558::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70304::DEBUG::2014-09-25 17:25:43,560::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:25:43,878::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:43,878::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:43,879::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:43,879::__init__::232::IOProcess::(_processLogs) (108152) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:43,879::__init__::232::IOProcess::(_processLogs) (108152) Queuing response >Thread-80::DEBUG::2014-09-25 17:25:43,880::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:25:43,890::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00054376 s, 695 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:43,891::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:43,891::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:43,892::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:43,892::__init__::232::IOProcess::(_processLogs) (108153) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:43,893::__init__::232::IOProcess::(_processLogs) (108153) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:44,335::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:44,336::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:44,336::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:44,336::__init__::232::IOProcess::(_processLogs) (108154) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:44,338::__init__::232::IOProcess::(_processLogs) (108154) Queuing response >Thread-79::DEBUG::2014-09-25 17:25:44,338::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:25:44,357::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0040103 s, 140 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:44,358::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:44,358::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:44,358::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:44,359::__init__::232::IOProcess::(_processLogs) (108155) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:44,361::__init__::232::IOProcess::(_processLogs) (108155) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:44,362::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:44,362::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:44,363::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:44,363::__init__::232::IOProcess::(_processLogs) (108156) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:44,368::__init__::232::IOProcess::(_processLogs) (108156) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:44,368::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:44,369::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:44,369::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:44,369::__init__::232::IOProcess::(_processLogs) (108157) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:44,373::__init__::232::IOProcess::(_processLogs) (108157) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:44,374::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:44,374::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:44,374::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:44,374::__init__::232::IOProcess::(_processLogs) (108158) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:44,379::__init__::232::IOProcess::(_processLogs) (108158) Queuing response >Dummy-148::DEBUG::2014-09-25 17:25:45,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:45,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0230741 s, 44.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:46,572::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:46,575::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70305::DEBUG::2014-09-25 17:25:46,581::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:47,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:47,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203818 s, 50.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:49,505::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:49,506::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:49,506::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:49,506::__init__::232::IOProcess::(_processLogs) (108159) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:49,508::__init__::232::IOProcess::(_processLogs) (108159) Queuing response >Thread-81::DEBUG::2014-09-25 17:25:49,509::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:25:49,518::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00073582 s, 504 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:49,519::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:49,520::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:49,520::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:49,520::__init__::232::IOProcess::(_processLogs) (108160) Got request for method 'statvfs' >Dummy-148::DEBUG::2014-09-25 17:25:49,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:25:49,530::__init__::232::IOProcess::(_processLogs) (108160) Queuing response >Dummy-148::DEBUG::2014-09-25 17:25:49,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175264 s, 58.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:49,595::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:49,597::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70306::DEBUG::2014-09-25 17:25:49,598::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:50,984::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70307::DEBUG::2014-09-25 17:25:50,985::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:25:50,985::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70307::DEBUG::2014-09-25 17:25:50,987::task::595::Storage.TaskManager.Task::(_updateState) Task=`4dbef5e8-0b8c-4ebe-b52c-638f1e453a2f`::moving from state init -> state preparing >Thread-70307::INFO::2014-09-25 17:25:50,988::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70307::INFO::2014-09-25 17:25:50,988::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70307::DEBUG::2014-09-25 17:25:50,989::task::1191::Storage.TaskManager.Task::(prepare) Task=`4dbef5e8-0b8c-4ebe-b52c-638f1e453a2f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70307::DEBUG::2014-09-25 17:25:50,989::task::595::Storage.TaskManager.Task::(_updateState) Task=`4dbef5e8-0b8c-4ebe-b52c-638f1e453a2f`::moving from state preparing -> state finished >Thread-70307::DEBUG::2014-09-25 17:25:50,989::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70307::DEBUG::2014-09-25 17:25:50,989::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70307::DEBUG::2014-09-25 17:25:50,989::task::993::Storage.TaskManager.Task::(_decref) Task=`4dbef5e8-0b8c-4ebe-b52c-638f1e453a2f`::ref 0 aborting False >Thread-70307::DEBUG::2014-09-25 17:25:50,990::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70307::DEBUG::2014-09-25 17:25:50,990::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:51,001::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70308::DEBUG::2014-09-25 17:25:51,003::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:25:51,003::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70308::DEBUG::2014-09-25 17:25:51,005::task::595::Storage.TaskManager.Task::(_updateState) Task=`ff6526cb-d124-4eb3-8038-4f3f6aebc029`::moving from state init -> state preparing >Thread-70308::INFO::2014-09-25 17:25:51,005::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70308::DEBUG::2014-09-25 17:25:51,006::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6d77d08c-d168-4042-b07e-e394ea076991`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70308::DEBUG::2014-09-25 17:25:51,006::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70308::DEBUG::2014-09-25 17:25:51,007::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70308::DEBUG::2014-09-25 17:25:51,007::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6d77d08c-d168-4042-b07e-e394ea076991`::Granted request >Thread-70308::DEBUG::2014-09-25 17:25:51,007::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ff6526cb-d124-4eb3-8038-4f3f6aebc029`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70308::DEBUG::2014-09-25 17:25:51,008::task::993::Storage.TaskManager.Task::(_decref) Task=`ff6526cb-d124-4eb3-8038-4f3f6aebc029`::ref 1 aborting False >Thread-70308::INFO::2014-09-25 17:25:51,012::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70308::DEBUG::2014-09-25 17:25:51,012::task::1191::Storage.TaskManager.Task::(prepare) Task=`ff6526cb-d124-4eb3-8038-4f3f6aebc029`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70308::DEBUG::2014-09-25 17:25:51,013::task::595::Storage.TaskManager.Task::(_updateState) Task=`ff6526cb-d124-4eb3-8038-4f3f6aebc029`::moving from state preparing -> state finished >Thread-70308::DEBUG::2014-09-25 17:25:51,013::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70308::DEBUG::2014-09-25 17:25:51,013::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70308::DEBUG::2014-09-25 17:25:51,013::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70308::DEBUG::2014-09-25 17:25:51,014::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70308::DEBUG::2014-09-25 17:25:51,014::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70308::DEBUG::2014-09-25 17:25:51,014::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70308::DEBUG::2014-09-25 17:25:51,015::task::993::Storage.TaskManager.Task::(_decref) Task=`ff6526cb-d124-4eb3-8038-4f3f6aebc029`::ref 0 aborting False >Thread-70308::DEBUG::2014-09-25 17:25:51,015::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70308::DEBUG::2014-09-25 17:25:51,015::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:51,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:51,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0325115 s, 31.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:52,612::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:52,614::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70309::DEBUG::2014-09-25 17:25:52,616::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:53,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:53,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0233846 s, 43.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:53,894::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:53,895::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:53,895::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:53,895::__init__::232::IOProcess::(_processLogs) (108161) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:53,896::__init__::232::IOProcess::(_processLogs) (108161) Queuing response >Thread-80::DEBUG::2014-09-25 17:25:53,897::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:25:53,906::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00047252 s, 800 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:53,907::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:53,907::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:53,908::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:53,909::__init__::232::IOProcess::(_processLogs) (108162) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:53,909::__init__::232::IOProcess::(_processLogs) (108162) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:54,381::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:54,381::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:54,381::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:54,381::__init__::232::IOProcess::(_processLogs) (108163) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:54,389::__init__::232::IOProcess::(_processLogs) (108163) Queuing response >Thread-79::DEBUG::2014-09-25 17:25:54,390::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:25:54,414::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00348071 s, 161 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:54,416::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:54,416::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:54,416::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:54,416::__init__::232::IOProcess::(_processLogs) (108164) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:54,421::__init__::232::IOProcess::(_processLogs) (108164) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:54,423::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:54,424::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:54,424::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:54,424::__init__::232::IOProcess::(_processLogs) (108165) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:54,424::__init__::232::IOProcess::(_processLogs) (108165) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:54,425::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:54,425::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:54,426::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:54,426::__init__::232::IOProcess::(_processLogs) (108166) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:54,426::__init__::232::IOProcess::(_processLogs) (108166) Queuing response >Thread-29::DEBUG::2014-09-25 17:25:54,427::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:54,428::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:54,429::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:54,429::__init__::232::IOProcess::(_processLogs) (108167) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:25:54,429::__init__::232::IOProcess::(_processLogs) (108167) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:55,630::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:55,632::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70310::DEBUG::2014-09-25 17:25:55,634::task::595::Storage.TaskManager.Task::(_updateState) Task=`6f14a157-b0de-448c-be1b-4b3172f86389`::moving from state init -> state preparing >Thread-70310::INFO::2014-09-25 17:25:55,634::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70310::INFO::2014-09-25 17:25:55,635::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00348071', 'lastCheck': '1.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00047252', 'lastCheck': '1.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00073582', 'lastCheck': '6.1', 'valid': True}} >Thread-70310::DEBUG::2014-09-25 17:25:55,635::task::1191::Storage.TaskManager.Task::(prepare) Task=`6f14a157-b0de-448c-be1b-4b3172f86389`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00348071', 'lastCheck': '1.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00047252', 'lastCheck': '1.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00073582', 'lastCheck': '6.1', 'valid': True}} >Thread-70310::DEBUG::2014-09-25 17:25:55,635::task::595::Storage.TaskManager.Task::(_updateState) Task=`6f14a157-b0de-448c-be1b-4b3172f86389`::moving from state preparing -> state finished >Thread-70310::DEBUG::2014-09-25 17:25:55,636::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70310::DEBUG::2014-09-25 17:25:55,636::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70310::DEBUG::2014-09-25 17:25:55,636::task::993::Storage.TaskManager.Task::(_decref) Task=`6f14a157-b0de-448c-be1b-4b3172f86389`::ref 0 aborting False >Thread-70310::DEBUG::2014-09-25 17:25:55,638::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:55,656::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:55,657::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70311::DEBUG::2014-09-25 17:25:55,659::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:25:55,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:55,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0299817 s, 34.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:25:57,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:57,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0217352 s, 47.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:25:58,703::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:25:58,705::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70312::DEBUG::2014-09-25 17:25:58,707::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:25:59,532::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:59,533::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:59,533::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:59,533::__init__::232::IOProcess::(_processLogs) (108168) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:59,556::__init__::232::IOProcess::(_processLogs) (108168) Queuing response >Thread-81::DEBUG::2014-09-25 17:25:59,558::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:25:59,598::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00311081 s, 119 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:25:59,599::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:25:59,599::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:25:59,599::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:25:59,600::__init__::232::IOProcess::(_processLogs) (108169) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:25:59,603::__init__::232::IOProcess::(_processLogs) (108169) Queuing response >Dummy-148::DEBUG::2014-09-25 17:25:59,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:25:59,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185501 s, 55.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:01,038::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70313::DEBUG::2014-09-25 17:26:01,039::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:26:01,039::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70313::DEBUG::2014-09-25 17:26:01,042::task::595::Storage.TaskManager.Task::(_updateState) Task=`03de7932-742a-4d3d-90ea-4cf801ecfbb9`::moving from state init -> state preparing >Thread-70313::INFO::2014-09-25 17:26:01,042::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70313::INFO::2014-09-25 17:26:01,043::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70313::DEBUG::2014-09-25 17:26:01,043::task::1191::Storage.TaskManager.Task::(prepare) Task=`03de7932-742a-4d3d-90ea-4cf801ecfbb9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70313::DEBUG::2014-09-25 17:26:01,043::task::595::Storage.TaskManager.Task::(_updateState) Task=`03de7932-742a-4d3d-90ea-4cf801ecfbb9`::moving from state preparing -> state finished >Thread-70313::DEBUG::2014-09-25 17:26:01,043::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70313::DEBUG::2014-09-25 17:26:01,044::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70313::DEBUG::2014-09-25 17:26:01,044::task::993::Storage.TaskManager.Task::(_decref) Task=`03de7932-742a-4d3d-90ea-4cf801ecfbb9`::ref 0 aborting False >Thread-70313::DEBUG::2014-09-25 17:26:01,044::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70313::DEBUG::2014-09-25 17:26:01,045::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:01,057::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:01,059::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70314::DEBUG::2014-09-25 17:26:01,059::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70314::DEBUG::2014-09-25 17:26:01,060::task::595::Storage.TaskManager.Task::(_updateState) Task=`d12ea872-919a-4bab-afd8-e35fb26b9be7`::moving from state init -> state preparing >Thread-70314::INFO::2014-09-25 17:26:01,061::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70314::DEBUG::2014-09-25 17:26:01,061::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`73ee7273-3b65-485c-9f4a-7525dd0138b8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70314::DEBUG::2014-09-25 17:26:01,062::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70314::DEBUG::2014-09-25 17:26:01,062::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70314::DEBUG::2014-09-25 17:26:01,062::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`73ee7273-3b65-485c-9f4a-7525dd0138b8`::Granted request >Thread-70314::DEBUG::2014-09-25 17:26:01,063::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`d12ea872-919a-4bab-afd8-e35fb26b9be7`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70314::DEBUG::2014-09-25 17:26:01,063::task::993::Storage.TaskManager.Task::(_decref) Task=`d12ea872-919a-4bab-afd8-e35fb26b9be7`::ref 1 aborting False >Thread-70314::INFO::2014-09-25 17:26:01,067::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70314::DEBUG::2014-09-25 17:26:01,068::task::1191::Storage.TaskManager.Task::(prepare) Task=`d12ea872-919a-4bab-afd8-e35fb26b9be7`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70314::DEBUG::2014-09-25 17:26:01,068::task::595::Storage.TaskManager.Task::(_updateState) Task=`d12ea872-919a-4bab-afd8-e35fb26b9be7`::moving from state preparing -> state finished >Thread-70314::DEBUG::2014-09-25 17:26:01,068::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70314::DEBUG::2014-09-25 17:26:01,068::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70314::DEBUG::2014-09-25 17:26:01,069::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70314::DEBUG::2014-09-25 17:26:01,069::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70314::DEBUG::2014-09-25 17:26:01,069::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70314::DEBUG::2014-09-25 17:26:01,069::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70314::DEBUG::2014-09-25 17:26:01,070::task::993::Storage.TaskManager.Task::(_decref) Task=`d12ea872-919a-4bab-afd8-e35fb26b9be7`::ref 0 aborting False >Thread-70314::DEBUG::2014-09-25 17:26:01,070::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70314::DEBUG::2014-09-25 17:26:01,071::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:01,722::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:01,723::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70315::DEBUG::2014-09-25 17:26:01,725::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:01,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:01,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0176382 s, 58.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:03,910::__init__::232::IOProcess::(_processLogs) Receiving request... >Dummy-148::DEBUG::2014-09-25 17:26:03,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:26:03,912::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-80::DEBUG::2014-09-25 17:26:03,921::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:26:03,922::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:03,930::__init__::232::IOProcess::(_processLogs) (108170) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:03,931::__init__::232::IOProcess::(_processLogs) (108170) Queuing response >Thread-80::DEBUG::2014-09-25 17:26:03,932::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000270331 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:03,933::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:03,934::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:03,934::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:03,936::__init__::232::IOProcess::(_processLogs) (108171) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:03,936::__init__::232::IOProcess::(_processLogs) (108171) Queuing response >Dummy-148::DEBUG::2014-09-25 17:26:03,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205618 s, 49.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:04,429::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:04,430::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:04,430::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:04,430::__init__::232::IOProcess::(_processLogs) (108172) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:04,431::__init__::232::IOProcess::(_processLogs) (108172) Queuing response >Thread-79::DEBUG::2014-09-25 17:26:04,432::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:26:04,449::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00313635 s, 179 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:04,450::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:04,450::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:04,450::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:04,451::__init__::232::IOProcess::(_processLogs) (108173) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:04,453::__init__::232::IOProcess::(_processLogs) (108173) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:04,454::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:04,454::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:04,454::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:04,455::__init__::232::IOProcess::(_processLogs) (108174) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:04,456::__init__::232::IOProcess::(_processLogs) (108174) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:04,456::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:04,456::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:04,457::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:04,457::__init__::232::IOProcess::(_processLogs) (108175) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:04,458::__init__::232::IOProcess::(_processLogs) (108175) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:04,458::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:04,458::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:04,459::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:04,460::__init__::232::IOProcess::(_processLogs) (108176) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:04,460::__init__::232::IOProcess::(_processLogs) (108176) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:04,737::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:04,739::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70316::DEBUG::2014-09-25 17:26:04,741::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:05,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:05,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134869 s, 75.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:07,754::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:07,755::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70317::DEBUG::2014-09-25 17:26:07,758::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:08,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:08,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186286 s, 55.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:09,605::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:09,605::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:09,605::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:09,606::__init__::232::IOProcess::(_processLogs) (108177) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:09,615::__init__::232::IOProcess::(_processLogs) (108177) Queuing response >Thread-81::DEBUG::2014-09-25 17:26:09,617::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:26:09,647::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0152758 s, 24.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:09,648::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:09,649::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:09,649::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:09,649::__init__::232::IOProcess::(_processLogs) (108178) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:09,661::__init__::232::IOProcess::(_processLogs) (108178) Queuing response >Dummy-148::DEBUG::2014-09-25 17:26:10,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:10,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143004 s, 71.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:10,770::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:10,772::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70318::DEBUG::2014-09-25 17:26:10,774::task::595::Storage.TaskManager.Task::(_updateState) Task=`304e7baf-bb30-4c61-bee8-d5576b769633`::moving from state init -> state preparing >Thread-70318::INFO::2014-09-25 17:26:10,775::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70318::INFO::2014-09-25 17:26:10,775::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00313635', 'lastCheck': '6.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000270331', 'lastCheck': '6.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0152758', 'lastCheck': '1.1', 'valid': True}} >Thread-70318::DEBUG::2014-09-25 17:26:10,776::task::1191::Storage.TaskManager.Task::(prepare) Task=`304e7baf-bb30-4c61-bee8-d5576b769633`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00313635', 'lastCheck': '6.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000270331', 'lastCheck': '6.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0152758', 'lastCheck': '1.1', 'valid': True}} >Thread-70318::DEBUG::2014-09-25 17:26:10,776::task::595::Storage.TaskManager.Task::(_updateState) Task=`304e7baf-bb30-4c61-bee8-d5576b769633`::moving from state preparing -> state finished >Thread-70318::DEBUG::2014-09-25 17:26:10,776::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70318::DEBUG::2014-09-25 17:26:10,776::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70318::DEBUG::2014-09-25 17:26:10,776::task::993::Storage.TaskManager.Task::(_decref) Task=`304e7baf-bb30-4c61-bee8-d5576b769633`::ref 0 aborting False >Thread-70318::DEBUG::2014-09-25 17:26:10,779::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:10,794::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:10,795::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70319::DEBUG::2014-09-25 17:26:10,797::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:11,103::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:11,104::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70320::DEBUG::2014-09-25 17:26:11,104::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70320::DEBUG::2014-09-25 17:26:11,106::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6b151b8-b3bb-4bc4-a577-7243d8977bd1`::moving from state init -> state preparing >Thread-70320::INFO::2014-09-25 17:26:11,106::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70320::INFO::2014-09-25 17:26:11,107::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70320::DEBUG::2014-09-25 17:26:11,107::task::1191::Storage.TaskManager.Task::(prepare) Task=`f6b151b8-b3bb-4bc4-a577-7243d8977bd1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70320::DEBUG::2014-09-25 17:26:11,107::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6b151b8-b3bb-4bc4-a577-7243d8977bd1`::moving from state preparing -> state finished >Thread-70320::DEBUG::2014-09-25 17:26:11,107::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70320::DEBUG::2014-09-25 17:26:11,107::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70320::DEBUG::2014-09-25 17:26:11,108::task::993::Storage.TaskManager.Task::(_decref) Task=`f6b151b8-b3bb-4bc4-a577-7243d8977bd1`::ref 0 aborting False >Thread-70320::DEBUG::2014-09-25 17:26:11,108::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70320::DEBUG::2014-09-25 17:26:11,108::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:11,119::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70321::DEBUG::2014-09-25 17:26:11,120::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:26:11,120::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70321::DEBUG::2014-09-25 17:26:11,123::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf78ff85-a5fc-4bda-bc27-ff09b7dbb596`::moving from state init -> state preparing >Thread-70321::INFO::2014-09-25 17:26:11,123::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70321::DEBUG::2014-09-25 17:26:11,124::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e0cf98dd-7749-449b-bfbe-4c9c481f17b1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70321::DEBUG::2014-09-25 17:26:11,124::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70321::DEBUG::2014-09-25 17:26:11,124::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70321::DEBUG::2014-09-25 17:26:11,124::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e0cf98dd-7749-449b-bfbe-4c9c481f17b1`::Granted request >Thread-70321::DEBUG::2014-09-25 17:26:11,125::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`bf78ff85-a5fc-4bda-bc27-ff09b7dbb596`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70321::DEBUG::2014-09-25 17:26:11,125::task::993::Storage.TaskManager.Task::(_decref) Task=`bf78ff85-a5fc-4bda-bc27-ff09b7dbb596`::ref 1 aborting False >Thread-70321::INFO::2014-09-25 17:26:11,133::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70321::DEBUG::2014-09-25 17:26:11,133::task::1191::Storage.TaskManager.Task::(prepare) Task=`bf78ff85-a5fc-4bda-bc27-ff09b7dbb596`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70321::DEBUG::2014-09-25 17:26:11,133::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf78ff85-a5fc-4bda-bc27-ff09b7dbb596`::moving from state preparing -> state finished >Thread-70321::DEBUG::2014-09-25 17:26:11,134::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70321::DEBUG::2014-09-25 17:26:11,134::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70321::DEBUG::2014-09-25 17:26:11,134::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70321::DEBUG::2014-09-25 17:26:11,135::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70321::DEBUG::2014-09-25 17:26:11,135::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70321::DEBUG::2014-09-25 17:26:11,135::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70321::DEBUG::2014-09-25 17:26:11,135::task::993::Storage.TaskManager.Task::(_decref) Task=`bf78ff85-a5fc-4bda-bc27-ff09b7dbb596`::ref 0 aborting False >Thread-70321::DEBUG::2014-09-25 17:26:11,136::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70321::DEBUG::2014-09-25 17:26:11,136::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:12,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:12,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0216724 s, 47.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:13,850::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:13,851::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70322::DEBUG::2014-09-25 17:26:13,853::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:26:13,936::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:13,937::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:13,937::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:13,937::__init__::232::IOProcess::(_processLogs) (108179) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:13,938::__init__::232::IOProcess::(_processLogs) (108179) Queuing response >Thread-80::DEBUG::2014-09-25 17:26:13,939::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:26:13,948::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000614661 s, 615 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:13,949::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:13,950::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:13,950::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:13,950::__init__::232::IOProcess::(_processLogs) (108180) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:13,950::__init__::232::IOProcess::(_processLogs) (108180) Queuing response >Dummy-148::DEBUG::2014-09-25 17:26:14,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:14,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016971 s, 60.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:14,461::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:14,461::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:14,461::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:14,462::__init__::232::IOProcess::(_processLogs) (108181) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:14,463::__init__::232::IOProcess::(_processLogs) (108181) Queuing response >Thread-79::DEBUG::2014-09-25 17:26:14,465::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:26:14,476::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00131827 s, 426 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:14,477::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:14,477::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:14,477::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:14,478::__init__::232::IOProcess::(_processLogs) (108182) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:14,482::__init__::232::IOProcess::(_processLogs) (108182) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:14,483::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:14,483::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:14,484::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:14,484::__init__::232::IOProcess::(_processLogs) (108183) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:14,485::__init__::232::IOProcess::(_processLogs) (108183) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:14,485::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:14,486::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:14,486::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:14,486::__init__::232::IOProcess::(_processLogs) (108184) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:14,486::__init__::232::IOProcess::(_processLogs) (108184) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:14,487::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:14,488::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:14,489::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:14,489::__init__::232::IOProcess::(_processLogs) (108185) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:14,489::__init__::232::IOProcess::(_processLogs) (108185) Queuing response >Dummy-148::DEBUG::2014-09-25 17:26:16,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:16,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0321704 s, 31.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:16,868::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:16,872::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70323::DEBUG::2014-09-25 17:26:16,877::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:18,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:18,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186881 s, 54.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:19,664::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:19,664::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:19,664::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:19,664::__init__::232::IOProcess::(_processLogs) (108186) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:19,672::__init__::232::IOProcess::(_processLogs) (108186) Queuing response >Thread-81::DEBUG::2014-09-25 17:26:19,673::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:26:19,709::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0125775 s, 29.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:19,710::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:19,710::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:19,711::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:19,711::__init__::232::IOProcess::(_processLogs) (108187) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:19,718::__init__::232::IOProcess::(_processLogs) (108187) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:19,892::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:19,894::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70324::DEBUG::2014-09-25 17:26:19,896::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:20,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:20,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213386 s, 48.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:21,162::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:21,163::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70325::DEBUG::2014-09-25 17:26:21,163::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70325::DEBUG::2014-09-25 17:26:21,165::task::595::Storage.TaskManager.Task::(_updateState) Task=`9d059973-0a81-4615-a010-aed2109adc63`::moving from state init -> state preparing >Thread-70325::INFO::2014-09-25 17:26:21,165::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70325::INFO::2014-09-25 17:26:21,166::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70325::DEBUG::2014-09-25 17:26:21,166::task::1191::Storage.TaskManager.Task::(prepare) Task=`9d059973-0a81-4615-a010-aed2109adc63`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70325::DEBUG::2014-09-25 17:26:21,166::task::595::Storage.TaskManager.Task::(_updateState) Task=`9d059973-0a81-4615-a010-aed2109adc63`::moving from state preparing -> state finished >Thread-70325::DEBUG::2014-09-25 17:26:21,167::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70325::DEBUG::2014-09-25 17:26:21,167::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70325::DEBUG::2014-09-25 17:26:21,167::task::993::Storage.TaskManager.Task::(_decref) Task=`9d059973-0a81-4615-a010-aed2109adc63`::ref 0 aborting False >Thread-70325::DEBUG::2014-09-25 17:26:21,167::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70325::DEBUG::2014-09-25 17:26:21,168::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:21,178::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:21,180::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70326::DEBUG::2014-09-25 17:26:21,180::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70326::DEBUG::2014-09-25 17:26:21,181::task::595::Storage.TaskManager.Task::(_updateState) Task=`d78a67e5-4e79-4ce3-bd6c-0e73e37dd874`::moving from state init -> state preparing >Thread-70326::INFO::2014-09-25 17:26:21,182::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70326::DEBUG::2014-09-25 17:26:21,182::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6e11a3eb-68bf-455b-8cf1-1db3dd823dc8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70326::DEBUG::2014-09-25 17:26:21,183::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70326::DEBUG::2014-09-25 17:26:21,183::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70326::DEBUG::2014-09-25 17:26:21,183::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6e11a3eb-68bf-455b-8cf1-1db3dd823dc8`::Granted request >Thread-70326::DEBUG::2014-09-25 17:26:21,184::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`d78a67e5-4e79-4ce3-bd6c-0e73e37dd874`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70326::DEBUG::2014-09-25 17:26:21,184::task::993::Storage.TaskManager.Task::(_decref) Task=`d78a67e5-4e79-4ce3-bd6c-0e73e37dd874`::ref 1 aborting False >Thread-70326::INFO::2014-09-25 17:26:21,189::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70326::DEBUG::2014-09-25 17:26:21,189::task::1191::Storage.TaskManager.Task::(prepare) Task=`d78a67e5-4e79-4ce3-bd6c-0e73e37dd874`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70326::DEBUG::2014-09-25 17:26:21,189::task::595::Storage.TaskManager.Task::(_updateState) Task=`d78a67e5-4e79-4ce3-bd6c-0e73e37dd874`::moving from state preparing -> state finished >Thread-70326::DEBUG::2014-09-25 17:26:21,189::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70326::DEBUG::2014-09-25 17:26:21,190::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70326::DEBUG::2014-09-25 17:26:21,190::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70326::DEBUG::2014-09-25 17:26:21,190::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70326::DEBUG::2014-09-25 17:26:21,190::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70326::DEBUG::2014-09-25 17:26:21,191::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70326::DEBUG::2014-09-25 17:26:21,191::task::993::Storage.TaskManager.Task::(_decref) Task=`d78a67e5-4e79-4ce3-bd6c-0e73e37dd874`::ref 0 aborting False >Thread-70326::DEBUG::2014-09-25 17:26:21,191::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70326::DEBUG::2014-09-25 17:26:21,192::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:22,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:22,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168522 s, 60.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:22,908::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:22,909::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70327::DEBUG::2014-09-25 17:26:22,912::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:26:23,953::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:23,953::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:23,954::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:23,954::__init__::232::IOProcess::(_processLogs) (108188) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:23,954::__init__::232::IOProcess::(_processLogs) (108188) Queuing response >Thread-80::DEBUG::2014-09-25 17:26:23,956::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:26:23,965::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000514336 s, 735 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:23,966::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:23,966::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:23,967::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:23,968::__init__::232::IOProcess::(_processLogs) (108189) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:23,968::__init__::232::IOProcess::(_processLogs) (108189) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:24,489::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:24,490::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:24,490::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:24,490::__init__::232::IOProcess::(_processLogs) (108190) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:24,494::__init__::232::IOProcess::(_processLogs) (108190) Queuing response >Thread-79::DEBUG::2014-09-25 17:26:24,495::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:26:24,512::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000929192 s, 605 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:24,513::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:24,514::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:24,514::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:24,514::__init__::232::IOProcess::(_processLogs) (108191) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:24,518::__init__::232::IOProcess::(_processLogs) (108191) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:24,519::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:24,520::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:24,520::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:24,520::__init__::232::IOProcess::(_processLogs) (108192) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:24,521::__init__::232::IOProcess::(_processLogs) (108192) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:24,521::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:24,521::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:24,522::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:24,522::__init__::232::IOProcess::(_processLogs) (108193) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:24,523::__init__::232::IOProcess::(_processLogs) (108193) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:24,523::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:24,523::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:24,524::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:24,525::__init__::232::IOProcess::(_processLogs) (108194) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:24,525::__init__::232::IOProcess::(_processLogs) (108194) Queuing response >Dummy-148::DEBUG::2014-09-25 17:26:24,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:24,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160367 s, 63.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:25,923::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:25,925::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70328::DEBUG::2014-09-25 17:26:25,927::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b405a6d-22ef-45fe-b45f-4b7a2c748d6e`::moving from state init -> state preparing >Thread-70328::INFO::2014-09-25 17:26:25,928::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70328::INFO::2014-09-25 17:26:25,928::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000929192', 'lastCheck': '1.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000514336', 'lastCheck': '2.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0125775', 'lastCheck': '6.2', 'valid': True}} >Thread-70328::DEBUG::2014-09-25 17:26:25,928::task::1191::Storage.TaskManager.Task::(prepare) Task=`4b405a6d-22ef-45fe-b45f-4b7a2c748d6e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000929192', 'lastCheck': '1.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000514336', 'lastCheck': '2.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0125775', 'lastCheck': '6.2', 'valid': True}} >Thread-70328::DEBUG::2014-09-25 17:26:25,929::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b405a6d-22ef-45fe-b45f-4b7a2c748d6e`::moving from state preparing -> state finished >Thread-70328::DEBUG::2014-09-25 17:26:25,929::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70328::DEBUG::2014-09-25 17:26:25,929::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70328::DEBUG::2014-09-25 17:26:25,929::task::993::Storage.TaskManager.Task::(_decref) Task=`4b405a6d-22ef-45fe-b45f-4b7a2c748d6e`::ref 0 aborting False >Thread-70328::DEBUG::2014-09-25 17:26:25,932::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:25,952::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:25,953::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70329::DEBUG::2014-09-25 17:26:25,954::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:26,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:26,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141715 s, 72.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:26:28,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:28,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138241 s, 74.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:29,032::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:29,034::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70330::DEBUG::2014-09-25 17:26:29,036::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:26:29,719::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:29,720::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:29,720::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:29,720::__init__::232::IOProcess::(_processLogs) (108195) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:29,728::__init__::232::IOProcess::(_processLogs) (108195) Queuing response >Thread-81::DEBUG::2014-09-25 17:26:29,728::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:26:29,745::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00729952 s, 50.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:29,746::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:29,746::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:29,746::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:29,747::__init__::232::IOProcess::(_processLogs) (108196) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:29,753::__init__::232::IOProcess::(_processLogs) (108196) Queuing response >Dummy-148::DEBUG::2014-09-25 17:26:30,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:30,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218292 s, 46.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:31,216::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:31,218::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70331::DEBUG::2014-09-25 17:26:31,218::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70331::DEBUG::2014-09-25 17:26:31,220::task::595::Storage.TaskManager.Task::(_updateState) Task=`980969d7-c2e3-414e-9452-4119df445372`::moving from state init -> state preparing >Thread-70331::INFO::2014-09-25 17:26:31,220::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70331::INFO::2014-09-25 17:26:31,221::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70331::DEBUG::2014-09-25 17:26:31,221::task::1191::Storage.TaskManager.Task::(prepare) Task=`980969d7-c2e3-414e-9452-4119df445372`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70331::DEBUG::2014-09-25 17:26:31,221::task::595::Storage.TaskManager.Task::(_updateState) Task=`980969d7-c2e3-414e-9452-4119df445372`::moving from state preparing -> state finished >Thread-70331::DEBUG::2014-09-25 17:26:31,222::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70331::DEBUG::2014-09-25 17:26:31,222::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70331::DEBUG::2014-09-25 17:26:31,222::task::993::Storage.TaskManager.Task::(_decref) Task=`980969d7-c2e3-414e-9452-4119df445372`::ref 0 aborting False >Thread-70331::DEBUG::2014-09-25 17:26:31,222::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70331::DEBUG::2014-09-25 17:26:31,223::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:31,234::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:31,235::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70332::DEBUG::2014-09-25 17:26:31,235::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70332::DEBUG::2014-09-25 17:26:31,237::task::595::Storage.TaskManager.Task::(_updateState) Task=`f1c02481-a091-40df-a97b-3faa9b9bc28d`::moving from state init -> state preparing >Thread-70332::INFO::2014-09-25 17:26:31,237::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70332::DEBUG::2014-09-25 17:26:31,239::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0b7b3618-4669-4447-a314-83323e2c7b3c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70332::DEBUG::2014-09-25 17:26:31,239::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70332::DEBUG::2014-09-25 17:26:31,239::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70332::DEBUG::2014-09-25 17:26:31,240::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0b7b3618-4669-4447-a314-83323e2c7b3c`::Granted request >Thread-70332::DEBUG::2014-09-25 17:26:31,240::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f1c02481-a091-40df-a97b-3faa9b9bc28d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70332::DEBUG::2014-09-25 17:26:31,240::task::993::Storage.TaskManager.Task::(_decref) Task=`f1c02481-a091-40df-a97b-3faa9b9bc28d`::ref 1 aborting False >Thread-70332::INFO::2014-09-25 17:26:31,245::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70332::DEBUG::2014-09-25 17:26:31,245::task::1191::Storage.TaskManager.Task::(prepare) Task=`f1c02481-a091-40df-a97b-3faa9b9bc28d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70332::DEBUG::2014-09-25 17:26:31,245::task::595::Storage.TaskManager.Task::(_updateState) Task=`f1c02481-a091-40df-a97b-3faa9b9bc28d`::moving from state preparing -> state finished >Thread-70332::DEBUG::2014-09-25 17:26:31,246::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70332::DEBUG::2014-09-25 17:26:31,246::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70332::DEBUG::2014-09-25 17:26:31,246::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70332::DEBUG::2014-09-25 17:26:31,246::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70332::DEBUG::2014-09-25 17:26:31,247::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70332::DEBUG::2014-09-25 17:26:31,247::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70332::DEBUG::2014-09-25 17:26:31,247::task::993::Storage.TaskManager.Task::(_decref) Task=`f1c02481-a091-40df-a97b-3faa9b9bc28d`::ref 0 aborting False >Thread-70332::DEBUG::2014-09-25 17:26:31,248::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70332::DEBUG::2014-09-25 17:26:31,248::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:32,048::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:32,049::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70333::DEBUG::2014-09-25 17:26:32,050::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:32,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:32,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212972 s, 48.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:33,969::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:33,969::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:33,970::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:33,970::__init__::232::IOProcess::(_processLogs) (108197) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:33,971::__init__::232::IOProcess::(_processLogs) (108197) Queuing response >Thread-80::DEBUG::2014-09-25 17:26:33,972::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:26:33,981::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00055139 s, 686 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:33,982::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:33,983::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:33,983::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:33,983::__init__::232::IOProcess::(_processLogs) (108198) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:33,984::__init__::232::IOProcess::(_processLogs) (108198) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:34,525::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:34,525::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:34,526::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:34,526::__init__::232::IOProcess::(_processLogs) (108199) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:34,528::__init__::232::IOProcess::(_processLogs) (108199) Queuing response >Thread-79::DEBUG::2014-09-25 17:26:34,529::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:26:34,547::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00234914 s, 239 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:34,549::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:34,549::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:34,549::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:34,549::__init__::232::IOProcess::(_processLogs) (108200) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:34,554::__init__::232::IOProcess::(_processLogs) (108200) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:34,556::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:34,556::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:34,557::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:34,557::__init__::232::IOProcess::(_processLogs) (108201) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:34,558::__init__::232::IOProcess::(_processLogs) (108201) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:34,558::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:34,559::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:34,559::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:34,559::__init__::232::IOProcess::(_processLogs) (108202) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:34,560::__init__::232::IOProcess::(_processLogs) (108202) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:34,560::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:34,561::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:34,562::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:34,562::__init__::232::IOProcess::(_processLogs) (108203) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:34,562::__init__::232::IOProcess::(_processLogs) (108203) Queuing response >Dummy-148::DEBUG::2014-09-25 17:26:34,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:34,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142145 s, 72.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:35,063::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:35,065::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70334::DEBUG::2014-09-25 17:26:35,067::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:36,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:36,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200272 s, 51.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:38,082::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:38,083::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70335::DEBUG::2014-09-25 17:26:38,085::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:38,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:38,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164414 s, 62.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:39,755::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:39,755::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:39,755::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:39,756::__init__::232::IOProcess::(_processLogs) (108204) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:39,758::__init__::232::IOProcess::(_processLogs) (108204) Queuing response >Thread-81::DEBUG::2014-09-25 17:26:39,759::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:26:39,774::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00424296 s, 87.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:39,775::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:39,775::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:39,776::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:39,776::__init__::232::IOProcess::(_processLogs) (108205) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:39,780::__init__::232::IOProcess::(_processLogs) (108205) Queuing response >Dummy-148::DEBUG::2014-09-25 17:26:41,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:41,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188083 s, 54.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:41,099::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:41,101::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70336::DEBUG::2014-09-25 17:26:41,104::task::595::Storage.TaskManager.Task::(_updateState) Task=`08e9153a-03bb-4087-8d76-fe0d8efa1ab0`::moving from state init -> state preparing >Thread-70336::INFO::2014-09-25 17:26:41,104::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70336::INFO::2014-09-25 17:26:41,104::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00234914', 'lastCheck': '6.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00055139', 'lastCheck': '7.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00424296', 'lastCheck': '1.3', 'valid': True}} >Thread-70336::DEBUG::2014-09-25 17:26:41,105::task::1191::Storage.TaskManager.Task::(prepare) Task=`08e9153a-03bb-4087-8d76-fe0d8efa1ab0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00234914', 'lastCheck': '6.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00055139', 'lastCheck': '7.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00424296', 'lastCheck': '1.3', 'valid': True}} >Thread-70336::DEBUG::2014-09-25 17:26:41,105::task::595::Storage.TaskManager.Task::(_updateState) Task=`08e9153a-03bb-4087-8d76-fe0d8efa1ab0`::moving from state preparing -> state finished >Thread-70336::DEBUG::2014-09-25 17:26:41,105::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70336::DEBUG::2014-09-25 17:26:41,105::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70336::DEBUG::2014-09-25 17:26:41,106::task::993::Storage.TaskManager.Task::(_decref) Task=`08e9153a-03bb-4087-8d76-fe0d8efa1ab0`::ref 0 aborting False >Thread-70336::DEBUG::2014-09-25 17:26:41,108::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:41,122::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:41,123::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70337::DEBUG::2014-09-25 17:26:41,125::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:41,272::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:41,273::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70338::DEBUG::2014-09-25 17:26:41,273::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70338::DEBUG::2014-09-25 17:26:41,275::task::595::Storage.TaskManager.Task::(_updateState) Task=`1129314d-42f0-4851-b1ca-a4a51e0a5090`::moving from state init -> state preparing >Thread-70338::INFO::2014-09-25 17:26:41,275::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70338::INFO::2014-09-25 17:26:41,276::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70338::DEBUG::2014-09-25 17:26:41,276::task::1191::Storage.TaskManager.Task::(prepare) Task=`1129314d-42f0-4851-b1ca-a4a51e0a5090`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70338::DEBUG::2014-09-25 17:26:41,276::task::595::Storage.TaskManager.Task::(_updateState) Task=`1129314d-42f0-4851-b1ca-a4a51e0a5090`::moving from state preparing -> state finished >Thread-70338::DEBUG::2014-09-25 17:26:41,276::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70338::DEBUG::2014-09-25 17:26:41,276::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70338::DEBUG::2014-09-25 17:26:41,277::task::993::Storage.TaskManager.Task::(_decref) Task=`1129314d-42f0-4851-b1ca-a4a51e0a5090`::ref 0 aborting False >Thread-70338::DEBUG::2014-09-25 17:26:41,277::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70338::DEBUG::2014-09-25 17:26:41,277::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:41,289::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:41,290::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70339::DEBUG::2014-09-25 17:26:41,290::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70339::DEBUG::2014-09-25 17:26:41,292::task::595::Storage.TaskManager.Task::(_updateState) Task=`8e33f4fe-d5aa-47e0-8ad9-3745794c56bf`::moving from state init -> state preparing >Thread-70339::INFO::2014-09-25 17:26:41,292::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70339::DEBUG::2014-09-25 17:26:41,293::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4ace64c4-3f8e-4719-bdca-a84d731c8caa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70339::DEBUG::2014-09-25 17:26:41,293::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70339::DEBUG::2014-09-25 17:26:41,293::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70339::DEBUG::2014-09-25 17:26:41,294::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4ace64c4-3f8e-4719-bdca-a84d731c8caa`::Granted request >Thread-70339::DEBUG::2014-09-25 17:26:41,294::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`8e33f4fe-d5aa-47e0-8ad9-3745794c56bf`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70339::DEBUG::2014-09-25 17:26:41,294::task::993::Storage.TaskManager.Task::(_decref) Task=`8e33f4fe-d5aa-47e0-8ad9-3745794c56bf`::ref 1 aborting False >Thread-70339::INFO::2014-09-25 17:26:41,299::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70339::DEBUG::2014-09-25 17:26:41,299::task::1191::Storage.TaskManager.Task::(prepare) Task=`8e33f4fe-d5aa-47e0-8ad9-3745794c56bf`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70339::DEBUG::2014-09-25 17:26:41,299::task::595::Storage.TaskManager.Task::(_updateState) Task=`8e33f4fe-d5aa-47e0-8ad9-3745794c56bf`::moving from state preparing -> state finished >Thread-70339::DEBUG::2014-09-25 17:26:41,300::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70339::DEBUG::2014-09-25 17:26:41,300::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70339::DEBUG::2014-09-25 17:26:41,300::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70339::DEBUG::2014-09-25 17:26:41,301::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70339::DEBUG::2014-09-25 17:26:41,301::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70339::DEBUG::2014-09-25 17:26:41,301::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70339::DEBUG::2014-09-25 17:26:41,302::task::993::Storage.TaskManager.Task::(_decref) Task=`8e33f4fe-d5aa-47e0-8ad9-3745794c56bf`::ref 0 aborting False >Thread-70339::DEBUG::2014-09-25 17:26:41,302::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70339::DEBUG::2014-09-25 17:26:41,302::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:43,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:43,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163574 s, 62.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:43,986::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:43,986::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:43,987::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:43,987::__init__::232::IOProcess::(_processLogs) (108206) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:43,987::__init__::232::IOProcess::(_processLogs) (108206) Queuing response >Thread-80::DEBUG::2014-09-25 17:26:43,988::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:26:43,997::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000436289 s, 866 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:43,999::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:43,999::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:43,999::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:44,000::__init__::232::IOProcess::(_processLogs) (108207) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:44,000::__init__::232::IOProcess::(_processLogs) (108207) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:44,171::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:44,172::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70340::DEBUG::2014-09-25 17:26:44,174::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:26:44,562::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:44,563::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:44,563::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:44,563::__init__::232::IOProcess::(_processLogs) (108208) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:44,567::__init__::232::IOProcess::(_processLogs) (108208) Queuing response >Thread-79::DEBUG::2014-09-25 17:26:44,568::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:26:44,587::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00208154 s, 270 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:44,588::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:44,589::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:44,589::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:44,589::__init__::232::IOProcess::(_processLogs) (108209) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:44,594::__init__::232::IOProcess::(_processLogs) (108209) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:44,596::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:44,596::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:44,596::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:44,597::__init__::232::IOProcess::(_processLogs) (108210) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:44,602::__init__::232::IOProcess::(_processLogs) (108210) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:44,603::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:44,604::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:44,604::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:44,604::__init__::232::IOProcess::(_processLogs) (108211) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:44,612::__init__::232::IOProcess::(_processLogs) (108211) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:44,613::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:44,613::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:44,613::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:44,614::__init__::232::IOProcess::(_processLogs) (108212) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:44,621::__init__::232::IOProcess::(_processLogs) (108212) Queuing response >Dummy-148::DEBUG::2014-09-25 17:26:45,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:45,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166203 s, 61.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:26:47,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:47,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0121203 s, 84.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:47,191::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:47,192::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70341::DEBUG::2014-09-25 17:26:47,194::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:49,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:49,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188471 s, 54.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:49,782::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:49,782::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:49,782::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:49,783::__init__::232::IOProcess::(_processLogs) (108213) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:49,784::__init__::232::IOProcess::(_processLogs) (108213) Queuing response >Thread-81::DEBUG::2014-09-25 17:26:49,785::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:26:49,805::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00638032 s, 58.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:49,806::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:49,806::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:49,806::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:49,807::__init__::232::IOProcess::(_processLogs) (108214) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:49,807::__init__::232::IOProcess::(_processLogs) (108214) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:50,209::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:50,210::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70342::DEBUG::2014-09-25 17:26:50,212::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:51,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:51,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.02394 s, 42.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:51,336::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:51,337::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70343::DEBUG::2014-09-25 17:26:51,337::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70343::DEBUG::2014-09-25 17:26:51,339::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ab0b85c-f8a8-4b45-a3ae-97f9758218f9`::moving from state init -> state preparing >Thread-70343::INFO::2014-09-25 17:26:51,340::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70343::INFO::2014-09-25 17:26:51,340::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70343::DEBUG::2014-09-25 17:26:51,340::task::1191::Storage.TaskManager.Task::(prepare) Task=`8ab0b85c-f8a8-4b45-a3ae-97f9758218f9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70343::DEBUG::2014-09-25 17:26:51,341::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ab0b85c-f8a8-4b45-a3ae-97f9758218f9`::moving from state preparing -> state finished >Thread-70343::DEBUG::2014-09-25 17:26:51,341::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70343::DEBUG::2014-09-25 17:26:51,341::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70343::DEBUG::2014-09-25 17:26:51,341::task::993::Storage.TaskManager.Task::(_decref) Task=`8ab0b85c-f8a8-4b45-a3ae-97f9758218f9`::ref 0 aborting False >Thread-70343::DEBUG::2014-09-25 17:26:51,342::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70343::DEBUG::2014-09-25 17:26:51,342::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:51,353::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70344::DEBUG::2014-09-25 17:26:51,354::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:26:51,354::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70344::DEBUG::2014-09-25 17:26:51,356::task::595::Storage.TaskManager.Task::(_updateState) Task=`46aaf0c2-ed07-4df0-8c3e-1a7aeb23e9cc`::moving from state init -> state preparing >Thread-70344::INFO::2014-09-25 17:26:51,356::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70344::DEBUG::2014-09-25 17:26:51,357::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c723055e-1ce1-47f0-a239-8b748a103c34`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70344::DEBUG::2014-09-25 17:26:51,357::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70344::DEBUG::2014-09-25 17:26:51,358::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70344::DEBUG::2014-09-25 17:26:51,358::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c723055e-1ce1-47f0-a239-8b748a103c34`::Granted request >Thread-70344::DEBUG::2014-09-25 17:26:51,358::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`46aaf0c2-ed07-4df0-8c3e-1a7aeb23e9cc`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70344::DEBUG::2014-09-25 17:26:51,359::task::993::Storage.TaskManager.Task::(_decref) Task=`46aaf0c2-ed07-4df0-8c3e-1a7aeb23e9cc`::ref 1 aborting False >Thread-70344::INFO::2014-09-25 17:26:51,363::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70344::DEBUG::2014-09-25 17:26:51,363::task::1191::Storage.TaskManager.Task::(prepare) Task=`46aaf0c2-ed07-4df0-8c3e-1a7aeb23e9cc`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70344::DEBUG::2014-09-25 17:26:51,364::task::595::Storage.TaskManager.Task::(_updateState) Task=`46aaf0c2-ed07-4df0-8c3e-1a7aeb23e9cc`::moving from state preparing -> state finished >Thread-70344::DEBUG::2014-09-25 17:26:51,364::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70344::DEBUG::2014-09-25 17:26:51,364::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70344::DEBUG::2014-09-25 17:26:51,364::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70344::DEBUG::2014-09-25 17:26:51,365::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70344::DEBUG::2014-09-25 17:26:51,365::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70344::DEBUG::2014-09-25 17:26:51,365::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70344::DEBUG::2014-09-25 17:26:51,365::task::993::Storage.TaskManager.Task::(_decref) Task=`46aaf0c2-ed07-4df0-8c3e-1a7aeb23e9cc`::ref 0 aborting False >Thread-70344::DEBUG::2014-09-25 17:26:51,366::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70344::DEBUG::2014-09-25 17:26:51,366::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:53,225::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:53,226::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70345::DEBUG::2014-09-25 17:26:53,228::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:53,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:53,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161114 s, 63.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:54,003::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:54,003::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:54,003::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:54,004::__init__::232::IOProcess::(_processLogs) (108215) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:54,004::__init__::232::IOProcess::(_processLogs) (108215) Queuing response >Thread-80::DEBUG::2014-09-25 17:26:54,005::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:26:54,014::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000580873 s, 651 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:54,015::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:54,016::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:54,016::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:54,016::__init__::232::IOProcess::(_processLogs) (108216) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:54,017::__init__::232::IOProcess::(_processLogs) (108216) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:54,622::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:54,623::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:54,623::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:54,623::__init__::232::IOProcess::(_processLogs) (108217) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:54,626::__init__::232::IOProcess::(_processLogs) (108217) Queuing response >Thread-79::DEBUG::2014-09-25 17:26:54,626::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:26:54,643::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00452365 s, 124 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:54,644::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:54,644::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:54,644::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:54,645::__init__::232::IOProcess::(_processLogs) (108218) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:54,649::__init__::232::IOProcess::(_processLogs) (108218) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:54,650::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:54,651::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:54,651::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:54,652::__init__::232::IOProcess::(_processLogs) (108219) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:54,652::__init__::232::IOProcess::(_processLogs) (108219) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:54,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:54,653::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:54,654::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:54,654::__init__::232::IOProcess::(_processLogs) (108220) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:54,654::__init__::232::IOProcess::(_processLogs) (108220) Queuing response >Thread-29::DEBUG::2014-09-25 17:26:54,654::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:54,655::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:54,655::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:54,657::__init__::232::IOProcess::(_processLogs) (108221) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:26:54,657::__init__::232::IOProcess::(_processLogs) (108221) Queuing response >Dummy-148::DEBUG::2014-09-25 17:26:55,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:55,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154609 s, 66.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:56,243::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:56,245::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70346::DEBUG::2014-09-25 17:26:56,247::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ebacbb6-67d0-4285-95e9-70ce24e2273a`::moving from state init -> state preparing >Thread-70346::INFO::2014-09-25 17:26:56,248::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70346::INFO::2014-09-25 17:26:56,248::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00452365', 'lastCheck': '1.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000580873', 'lastCheck': '2.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00638032', 'lastCheck': '6.4', 'valid': True}} >Thread-70346::DEBUG::2014-09-25 17:26:56,248::task::1191::Storage.TaskManager.Task::(prepare) Task=`7ebacbb6-67d0-4285-95e9-70ce24e2273a`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00452365', 'lastCheck': '1.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000580873', 'lastCheck': '2.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00638032', 'lastCheck': '6.4', 'valid': True}} >Thread-70346::DEBUG::2014-09-25 17:26:56,249::task::595::Storage.TaskManager.Task::(_updateState) Task=`7ebacbb6-67d0-4285-95e9-70ce24e2273a`::moving from state preparing -> state finished >Thread-70346::DEBUG::2014-09-25 17:26:56,249::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70346::DEBUG::2014-09-25 17:26:56,249::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70346::DEBUG::2014-09-25 17:26:56,249::task::993::Storage.TaskManager.Task::(_decref) Task=`7ebacbb6-67d0-4285-95e9-70ce24e2273a`::ref 0 aborting False >Thread-70346::DEBUG::2014-09-25 17:26:56,252::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:56,273::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:56,274::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70347::DEBUG::2014-09-25 17:26:56,276::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:57,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:57,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177495 s, 57.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:26:59,340::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:26:59,341::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70348::DEBUG::2014-09-25 17:26:59,343::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:26:59,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:26:59,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131945 s, 77.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:59,810::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:59,810::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:59,811::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:59,811::__init__::232::IOProcess::(_processLogs) (108222) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:59,812::__init__::232::IOProcess::(_processLogs) (108222) Queuing response >Thread-81::DEBUG::2014-09-25 17:26:59,813::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:26:59,823::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000344512 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:26:59,825::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:26:59,825::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:26:59,825::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:26:59,826::__init__::232::IOProcess::(_processLogs) (108223) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:26:59,827::__init__::232::IOProcess::(_processLogs) (108223) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:01,391::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70349::DEBUG::2014-09-25 17:27:01,392::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:27:01,392::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70349::DEBUG::2014-09-25 17:27:01,395::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d79e8a7-d2a3-4243-8795-0d69e4e9fb4b`::moving from state init -> state preparing >Thread-70349::INFO::2014-09-25 17:27:01,395::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70349::INFO::2014-09-25 17:27:01,395::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70349::DEBUG::2014-09-25 17:27:01,396::task::1191::Storage.TaskManager.Task::(prepare) Task=`8d79e8a7-d2a3-4243-8795-0d69e4e9fb4b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70349::DEBUG::2014-09-25 17:27:01,396::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d79e8a7-d2a3-4243-8795-0d69e4e9fb4b`::moving from state preparing -> state finished >Thread-70349::DEBUG::2014-09-25 17:27:01,396::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70349::DEBUG::2014-09-25 17:27:01,396::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70349::DEBUG::2014-09-25 17:27:01,397::task::993::Storage.TaskManager.Task::(_decref) Task=`8d79e8a7-d2a3-4243-8795-0d69e4e9fb4b`::ref 0 aborting False >Thread-70349::DEBUG::2014-09-25 17:27:01,397::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70349::DEBUG::2014-09-25 17:27:01,397::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:01,407::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:01,409::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70350::DEBUG::2014-09-25 17:27:01,409::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70350::DEBUG::2014-09-25 17:27:01,410::task::595::Storage.TaskManager.Task::(_updateState) Task=`94c90208-943f-4802-a4dd-139695bac844`::moving from state init -> state preparing >Thread-70350::INFO::2014-09-25 17:27:01,411::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70350::DEBUG::2014-09-25 17:27:01,411::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`dc664218-ebce-4be7-8ad8-819598eca3fb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70350::DEBUG::2014-09-25 17:27:01,412::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70350::DEBUG::2014-09-25 17:27:01,412::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70350::DEBUG::2014-09-25 17:27:01,412::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`dc664218-ebce-4be7-8ad8-819598eca3fb`::Granted request >Thread-70350::DEBUG::2014-09-25 17:27:01,413::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`94c90208-943f-4802-a4dd-139695bac844`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70350::DEBUG::2014-09-25 17:27:01,413::task::993::Storage.TaskManager.Task::(_decref) Task=`94c90208-943f-4802-a4dd-139695bac844`::ref 1 aborting False >Thread-70350::INFO::2014-09-25 17:27:01,417::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70350::DEBUG::2014-09-25 17:27:01,418::task::1191::Storage.TaskManager.Task::(prepare) Task=`94c90208-943f-4802-a4dd-139695bac844`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70350::DEBUG::2014-09-25 17:27:01,418::task::595::Storage.TaskManager.Task::(_updateState) Task=`94c90208-943f-4802-a4dd-139695bac844`::moving from state preparing -> state finished >Thread-70350::DEBUG::2014-09-25 17:27:01,418::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70350::DEBUG::2014-09-25 17:27:01,419::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70350::DEBUG::2014-09-25 17:27:01,419::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70350::DEBUG::2014-09-25 17:27:01,419::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70350::DEBUG::2014-09-25 17:27:01,419::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70350::DEBUG::2014-09-25 17:27:01,420::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70350::DEBUG::2014-09-25 17:27:01,420::task::993::Storage.TaskManager.Task::(_decref) Task=`94c90208-943f-4802-a4dd-139695bac844`::ref 0 aborting False >Thread-70350::DEBUG::2014-09-25 17:27:01,420::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70350::DEBUG::2014-09-25 17:27:01,421::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:01,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:01,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.030873 s, 33.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:02,358::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:02,359::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70351::DEBUG::2014-09-25 17:27:02,361::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:03,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:03,601::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161357 s, 63.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:04,019::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:04,020::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:04,020::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:04,020::__init__::232::IOProcess::(_processLogs) (108224) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:04,021::__init__::232::IOProcess::(_processLogs) (108224) Queuing response >Thread-80::DEBUG::2014-09-25 17:27:04,022::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:27:04,031::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000528208 s, 716 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:04,032::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:04,033::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:04,033::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:04,033::__init__::232::IOProcess::(_processLogs) (108225) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:04,034::__init__::232::IOProcess::(_processLogs) (108225) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:04,657::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:04,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:04,658::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:04,658::__init__::232::IOProcess::(_processLogs) (108226) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:04,659::__init__::232::IOProcess::(_processLogs) (108226) Queuing response >Thread-79::DEBUG::2014-09-25 17:27:04,660::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:27:04,683::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00588917 s, 95.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:04,684::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:04,684::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:04,685::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:04,685::__init__::232::IOProcess::(_processLogs) (108227) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:04,685::__init__::232::IOProcess::(_processLogs) (108227) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:04,686::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:04,687::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:04,688::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:04,688::__init__::232::IOProcess::(_processLogs) (108228) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:04,688::__init__::232::IOProcess::(_processLogs) (108228) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:04,689::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:04,689::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:04,689::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:04,690::__init__::232::IOProcess::(_processLogs) (108229) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:04,690::__init__::232::IOProcess::(_processLogs) (108229) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:04,691::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:04,691::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:04,692::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:04,693::__init__::232::IOProcess::(_processLogs) (108230) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:04,693::__init__::232::IOProcess::(_processLogs) (108230) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:05,374::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:05,375::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70352::DEBUG::2014-09-25 17:27:05,377::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:05,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:05,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214 s, 47.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:27:07,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:07,710::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181883 s, 56.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:08,390::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:08,392::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70353::DEBUG::2014-09-25 17:27:08,393::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:09,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:09,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134221 s, 76.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:09,828::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:09,828::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:09,828::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:09,829::__init__::232::IOProcess::(_processLogs) (108231) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:09,835::__init__::232::IOProcess::(_processLogs) (108231) Queuing response >Thread-81::DEBUG::2014-09-25 17:27:09,835::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:27:09,852::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00595065 s, 62.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:09,854::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:09,854::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:09,854::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:09,854::__init__::232::IOProcess::(_processLogs) (108232) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:09,858::__init__::232::IOProcess::(_processLogs) (108232) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:11,406::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:11,408::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70354::DEBUG::2014-09-25 17:27:11,410::task::595::Storage.TaskManager.Task::(_updateState) Task=`5900ab17-629b-4dfe-82cb-b26265b937fe`::moving from state init -> state preparing >Thread-70354::INFO::2014-09-25 17:27:11,411::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70354::INFO::2014-09-25 17:27:11,411::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00588917', 'lastCheck': '6.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000528208', 'lastCheck': '7.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00595065', 'lastCheck': '1.6', 'valid': True}} >Thread-70354::DEBUG::2014-09-25 17:27:11,412::task::1191::Storage.TaskManager.Task::(prepare) Task=`5900ab17-629b-4dfe-82cb-b26265b937fe`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00588917', 'lastCheck': '6.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000528208', 'lastCheck': '7.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00595065', 'lastCheck': '1.6', 'valid': True}} >Thread-70354::DEBUG::2014-09-25 17:27:11,412::task::595::Storage.TaskManager.Task::(_updateState) Task=`5900ab17-629b-4dfe-82cb-b26265b937fe`::moving from state preparing -> state finished >Thread-70354::DEBUG::2014-09-25 17:27:11,412::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70354::DEBUG::2014-09-25 17:27:11,412::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70354::DEBUG::2014-09-25 17:27:11,412::task::993::Storage.TaskManager.Task::(_decref) Task=`5900ab17-629b-4dfe-82cb-b26265b937fe`::ref 0 aborting False >Thread-70354::DEBUG::2014-09-25 17:27:11,415::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:11,437::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:11,438::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70355::DEBUG::2014-09-25 17:27:11,440::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:11,446::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:11,447::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70356::DEBUG::2014-09-25 17:27:11,447::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70356::DEBUG::2014-09-25 17:27:11,449::task::595::Storage.TaskManager.Task::(_updateState) Task=`79c0c3db-2e15-4ea3-8cac-c7eb7d52efe6`::moving from state init -> state preparing >Thread-70356::INFO::2014-09-25 17:27:11,449::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70356::INFO::2014-09-25 17:27:11,450::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70356::DEBUG::2014-09-25 17:27:11,450::task::1191::Storage.TaskManager.Task::(prepare) Task=`79c0c3db-2e15-4ea3-8cac-c7eb7d52efe6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70356::DEBUG::2014-09-25 17:27:11,450::task::595::Storage.TaskManager.Task::(_updateState) Task=`79c0c3db-2e15-4ea3-8cac-c7eb7d52efe6`::moving from state preparing -> state finished >Thread-70356::DEBUG::2014-09-25 17:27:11,450::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70356::DEBUG::2014-09-25 17:27:11,451::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70356::DEBUG::2014-09-25 17:27:11,451::task::993::Storage.TaskManager.Task::(_decref) Task=`79c0c3db-2e15-4ea3-8cac-c7eb7d52efe6`::ref 0 aborting False >Thread-70356::DEBUG::2014-09-25 17:27:11,451::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70356::DEBUG::2014-09-25 17:27:11,451::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:11,476::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:11,477::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70357::DEBUG::2014-09-25 17:27:11,477::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70357::DEBUG::2014-09-25 17:27:11,479::task::595::Storage.TaskManager.Task::(_updateState) Task=`6654a0cf-52d3-4637-96d7-6af79f444c55`::moving from state init -> state preparing >Thread-70357::INFO::2014-09-25 17:27:11,479::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70357::DEBUG::2014-09-25 17:27:11,480::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`da430156-faee-4558-8851-21f99a4401e1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70357::DEBUG::2014-09-25 17:27:11,480::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70357::DEBUG::2014-09-25 17:27:11,480::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70357::DEBUG::2014-09-25 17:27:11,480::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`da430156-faee-4558-8851-21f99a4401e1`::Granted request >Thread-70357::DEBUG::2014-09-25 17:27:11,482::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`6654a0cf-52d3-4637-96d7-6af79f444c55`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70357::DEBUG::2014-09-25 17:27:11,482::task::993::Storage.TaskManager.Task::(_decref) Task=`6654a0cf-52d3-4637-96d7-6af79f444c55`::ref 1 aborting False >Thread-70357::INFO::2014-09-25 17:27:11,487::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70357::DEBUG::2014-09-25 17:27:11,487::task::1191::Storage.TaskManager.Task::(prepare) Task=`6654a0cf-52d3-4637-96d7-6af79f444c55`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70357::DEBUG::2014-09-25 17:27:11,487::task::595::Storage.TaskManager.Task::(_updateState) Task=`6654a0cf-52d3-4637-96d7-6af79f444c55`::moving from state preparing -> state finished >Thread-70357::DEBUG::2014-09-25 17:27:11,488::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70357::DEBUG::2014-09-25 17:27:11,488::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70357::DEBUG::2014-09-25 17:27:11,488::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70357::DEBUG::2014-09-25 17:27:11,488::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70357::DEBUG::2014-09-25 17:27:11,489::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70357::DEBUG::2014-09-25 17:27:11,489::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70357::DEBUG::2014-09-25 17:27:11,489::task::993::Storage.TaskManager.Task::(_decref) Task=`6654a0cf-52d3-4637-96d7-6af79f444c55`::ref 0 aborting False >Thread-70357::DEBUG::2014-09-25 17:27:11,490::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70357::DEBUG::2014-09-25 17:27:11,490::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:11,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:11,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0190494 s, 53.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:27:13,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:13,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132243 s, 77.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:14,036::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:14,037::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:14,037::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:14,037::__init__::232::IOProcess::(_processLogs) (108233) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:14,038::__init__::232::IOProcess::(_processLogs) (108233) Queuing response >Thread-80::DEBUG::2014-09-25 17:27:14,039::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:27:14,048::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000537813 s, 703 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:14,049::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:14,049::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:14,050::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:14,050::__init__::232::IOProcess::(_processLogs) (108234) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:14,051::__init__::232::IOProcess::(_processLogs) (108234) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:14,515::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:14,516::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70358::DEBUG::2014-09-25 17:27:14,518::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:27:14,693::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:14,693::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:14,693::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:14,694::__init__::232::IOProcess::(_processLogs) (108235) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:14,694::__init__::232::IOProcess::(_processLogs) (108235) Queuing response >Thread-79::DEBUG::2014-09-25 17:27:14,695::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:27:14,707::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00152372 s, 369 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:14,708::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:14,709::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:14,709::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:14,709::__init__::232::IOProcess::(_processLogs) (108236) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:14,715::__init__::232::IOProcess::(_processLogs) (108236) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:14,716::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:14,717::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:14,718::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:14,718::__init__::232::IOProcess::(_processLogs) (108237) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:14,718::__init__::232::IOProcess::(_processLogs) (108237) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:14,719::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:14,719::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:14,720::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:14,720::__init__::232::IOProcess::(_processLogs) (108238) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:14,720::__init__::232::IOProcess::(_processLogs) (108238) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:14,721::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:14,721::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:14,722::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:14,723::__init__::232::IOProcess::(_processLogs) (108239) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:14,723::__init__::232::IOProcess::(_processLogs) (108239) Queuing response >Dummy-148::DEBUG::2014-09-25 17:27:15,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:15,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174406 s, 58.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:17,531::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:17,532::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70359::DEBUG::2014-09-25 17:27:17,534::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:17,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:17,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146565 s, 69.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:19,860::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:19,860::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:19,861::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:19,861::__init__::232::IOProcess::(_processLogs) (108240) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:19,866::__init__::232::IOProcess::(_processLogs) (108240) Queuing response >Thread-81::DEBUG::2014-09-25 17:27:19,866::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:27:19,883::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00294704 s, 126 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:19,884::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:19,884::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:19,884::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:19,885::__init__::232::IOProcess::(_processLogs) (108241) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:19,890::__init__::232::IOProcess::(_processLogs) (108241) Queuing response >Dummy-148::DEBUG::2014-09-25 17:27:19,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:19,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157581 s, 65.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:20,551::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:20,553::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70360::DEBUG::2014-09-25 17:27:20,555::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:21,532::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70361::DEBUG::2014-09-25 17:27:21,533::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:27:21,533::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70361::DEBUG::2014-09-25 17:27:21,535::task::595::Storage.TaskManager.Task::(_updateState) Task=`cc9f7128-44c3-475f-881a-b468f822e897`::moving from state init -> state preparing >Thread-70361::INFO::2014-09-25 17:27:21,536::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70361::INFO::2014-09-25 17:27:21,536::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70361::DEBUG::2014-09-25 17:27:21,536::task::1191::Storage.TaskManager.Task::(prepare) Task=`cc9f7128-44c3-475f-881a-b468f822e897`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70361::DEBUG::2014-09-25 17:27:21,537::task::595::Storage.TaskManager.Task::(_updateState) Task=`cc9f7128-44c3-475f-881a-b468f822e897`::moving from state preparing -> state finished >Thread-70361::DEBUG::2014-09-25 17:27:21,537::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70361::DEBUG::2014-09-25 17:27:21,537::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70361::DEBUG::2014-09-25 17:27:21,537::task::993::Storage.TaskManager.Task::(_decref) Task=`cc9f7128-44c3-475f-881a-b468f822e897`::ref 0 aborting False >Thread-70361::DEBUG::2014-09-25 17:27:21,538::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70361::DEBUG::2014-09-25 17:27:21,538::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:21,559::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:21,560::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70362::DEBUG::2014-09-25 17:27:21,560::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70362::DEBUG::2014-09-25 17:27:21,562::task::595::Storage.TaskManager.Task::(_updateState) Task=`170b9a9e-2a23-4778-a608-9845f6b45da3`::moving from state init -> state preparing >Thread-70362::INFO::2014-09-25 17:27:21,562::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70362::DEBUG::2014-09-25 17:27:21,563::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`670876ea-b175-47e6-adbb-de03447ec70a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70362::DEBUG::2014-09-25 17:27:21,563::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70362::DEBUG::2014-09-25 17:27:21,564::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70362::DEBUG::2014-09-25 17:27:21,564::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`670876ea-b175-47e6-adbb-de03447ec70a`::Granted request >Thread-70362::DEBUG::2014-09-25 17:27:21,564::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`170b9a9e-2a23-4778-a608-9845f6b45da3`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70362::DEBUG::2014-09-25 17:27:21,565::task::993::Storage.TaskManager.Task::(_decref) Task=`170b9a9e-2a23-4778-a608-9845f6b45da3`::ref 1 aborting False >Thread-70362::INFO::2014-09-25 17:27:21,569::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70362::DEBUG::2014-09-25 17:27:21,569::task::1191::Storage.TaskManager.Task::(prepare) Task=`170b9a9e-2a23-4778-a608-9845f6b45da3`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70362::DEBUG::2014-09-25 17:27:21,569::task::595::Storage.TaskManager.Task::(_updateState) Task=`170b9a9e-2a23-4778-a608-9845f6b45da3`::moving from state preparing -> state finished >Thread-70362::DEBUG::2014-09-25 17:27:21,570::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70362::DEBUG::2014-09-25 17:27:21,570::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70362::DEBUG::2014-09-25 17:27:21,570::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70362::DEBUG::2014-09-25 17:27:21,571::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70362::DEBUG::2014-09-25 17:27:21,571::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70362::DEBUG::2014-09-25 17:27:21,571::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70362::DEBUG::2014-09-25 17:27:21,571::task::993::Storage.TaskManager.Task::(_decref) Task=`170b9a9e-2a23-4778-a608-9845f6b45da3`::ref 0 aborting False >Thread-70362::DEBUG::2014-09-25 17:27:21,572::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70362::DEBUG::2014-09-25 17:27:21,572::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:22,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:22,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148968 s, 68.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:23,571::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:23,573::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70363::DEBUG::2014-09-25 17:27:23,575::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:27:24,053::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:24,053::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:24,053::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:24,053::__init__::232::IOProcess::(_processLogs) (108242) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:24,054::__init__::232::IOProcess::(_processLogs) (108242) Queuing response >Thread-80::DEBUG::2014-09-25 17:27:24,055::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:24,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-80::DEBUG::2014-09-25 17:27:24,071::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000515703 s, 733 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:24,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:24,072::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:24,073::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:24,073::__init__::232::IOProcess::(_processLogs) (108243) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:24,074::__init__::232::IOProcess::(_processLogs) (108243) Queuing response >Dummy-148::DEBUG::2014-09-25 17:27:24,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0222099 s, 46.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:24,723::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:24,724::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:24,724::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:24,725::__init__::232::IOProcess::(_processLogs) (108244) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 17:27:24,725::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:27:24,726::__init__::232::IOProcess::(_processLogs) (108244) Queuing response >Thread-79::DEBUG::2014-09-25 17:27:24,736::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000438916 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:24,737::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:24,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:24,737::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:24,737::__init__::232::IOProcess::(_processLogs) (108245) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:24,738::__init__::232::IOProcess::(_processLogs) (108245) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:24,740::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:24,740::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:24,741::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:24,741::__init__::232::IOProcess::(_processLogs) (108246) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:24,741::__init__::232::IOProcess::(_processLogs) (108246) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:24,742::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:24,742::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:24,742::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:24,742::__init__::232::IOProcess::(_processLogs) (108247) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:24,743::__init__::232::IOProcess::(_processLogs) (108247) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:24,744::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:24,744::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:24,744::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:24,745::__init__::232::IOProcess::(_processLogs) (108248) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:24,746::__init__::232::IOProcess::(_processLogs) (108248) Queuing response >Dummy-148::DEBUG::2014-09-25 17:27:26,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:26,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160732 s, 63.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:26,589::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:26,590::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70364::DEBUG::2014-09-25 17:27:26,593::task::595::Storage.TaskManager.Task::(_updateState) Task=`68142fe9-e5ce-4c0b-a1ec-bcbab43942d9`::moving from state init -> state preparing >Thread-70364::INFO::2014-09-25 17:27:26,594::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70364::INFO::2014-09-25 17:27:26,594::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000438916', 'lastCheck': '1.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000515703', 'lastCheck': '2.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00294704', 'lastCheck': '6.7', 'valid': True}} >Thread-70364::DEBUG::2014-09-25 17:27:26,594::task::1191::Storage.TaskManager.Task::(prepare) Task=`68142fe9-e5ce-4c0b-a1ec-bcbab43942d9`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000438916', 'lastCheck': '1.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000515703', 'lastCheck': '2.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00294704', 'lastCheck': '6.7', 'valid': True}} >Thread-70364::DEBUG::2014-09-25 17:27:26,595::task::595::Storage.TaskManager.Task::(_updateState) Task=`68142fe9-e5ce-4c0b-a1ec-bcbab43942d9`::moving from state preparing -> state finished >Thread-70364::DEBUG::2014-09-25 17:27:26,595::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70364::DEBUG::2014-09-25 17:27:26,595::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70364::DEBUG::2014-09-25 17:27:26,595::task::993::Storage.TaskManager.Task::(_decref) Task=`68142fe9-e5ce-4c0b-a1ec-bcbab43942d9`::ref 0 aborting False >Thread-70364::DEBUG::2014-09-25 17:27:26,598::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:26,617::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:26,618::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70365::DEBUG::2014-09-25 17:27:26,620::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:28,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:28,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211758 s, 48.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:29,677::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:29,679::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70366::DEBUG::2014-09-25 17:27:29,681::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 17:27:29,891::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:27:29,895::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:29,895::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:29,896::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:29,896::__init__::232::IOProcess::(_processLogs) (108249) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:27:29,896::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:29,897::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:29,897::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:29,897::__init__::232::IOProcess::(_processLogs) (108250) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:27:29,897::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:29,898::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:29,898::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:29,898::__init__::232::IOProcess::(_processLogs) (108251) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:27:29,899::__init__::232::IOProcess::(_processLogs) (108250) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:29,899::__init__::232::IOProcess::(_processLogs) (108249) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:29,902::__init__::232::IOProcess::(_processLogs) (108251) Queuing response >Thread-81::DEBUG::2014-09-25 17:27:29,905::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:27:29,906::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:29,906::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:29,907::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:29,907::__init__::232::IOProcess::(_processLogs) (108252) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:27:29,932::__init__::232::IOProcess::(_processLogs) (108252) Queuing response >Thread-81::DEBUG::2014-09-25 17:27:29,933::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:27:29,934::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:29,934::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:29,934::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:29,934::__init__::232::IOProcess::(_processLogs) (108253) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:27:29,941::__init__::232::IOProcess::(_processLogs) (108253) Queuing response >Thread-81::DEBUG::2014-09-25 17:27:29,941::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:27:29,942::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:29,943::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:29,943::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:29,944::__init__::232::IOProcess::(_processLogs) (108254) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:29,944::__init__::232::IOProcess::(_processLogs) (108254) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:29,944::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:29,945::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:29,945::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:29,945::__init__::232::IOProcess::(_processLogs) (108255) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:27:29,949::__init__::232::IOProcess::(_processLogs) (108255) Queuing response >Thread-81::DEBUG::2014-09-25 17:27:29,950::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 17:27:29,950::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:27:29,950::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:27:29,951::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:29,951::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:29,951::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:29,952::__init__::232::IOProcess::(_processLogs) (108256) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:29,953::__init__::232::IOProcess::(_processLogs) (108256) Queuing response >Thread-81::DEBUG::2014-09-25 17:27:29,954::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:27:29,966::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00200686 s, 185 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:29,967::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:29,967::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:29,968::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:29,968::__init__::232::IOProcess::(_processLogs) (108257) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:29,972::__init__::232::IOProcess::(_processLogs) (108257) Queuing response >Dummy-148::DEBUG::2014-09-25 17:27:30,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:30,265::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205995 s, 49.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:31,617::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70370::DEBUG::2014-09-25 17:27:31,618::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:27:31,618::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70370::DEBUG::2014-09-25 17:27:31,621::task::595::Storage.TaskManager.Task::(_updateState) Task=`49621650-0bae-412a-bcec-d078248c0f36`::moving from state init -> state preparing >Thread-70370::INFO::2014-09-25 17:27:31,621::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70370::INFO::2014-09-25 17:27:31,622::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70370::DEBUG::2014-09-25 17:27:31,622::task::1191::Storage.TaskManager.Task::(prepare) Task=`49621650-0bae-412a-bcec-d078248c0f36`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70370::DEBUG::2014-09-25 17:27:31,622::task::595::Storage.TaskManager.Task::(_updateState) Task=`49621650-0bae-412a-bcec-d078248c0f36`::moving from state preparing -> state finished >Thread-70370::DEBUG::2014-09-25 17:27:31,623::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70370::DEBUG::2014-09-25 17:27:31,623::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70370::DEBUG::2014-09-25 17:27:31,623::task::993::Storage.TaskManager.Task::(_decref) Task=`49621650-0bae-412a-bcec-d078248c0f36`::ref 0 aborting False >Thread-70370::DEBUG::2014-09-25 17:27:31,623::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70370::DEBUG::2014-09-25 17:27:31,624::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:31,638::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:31,641::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70371::DEBUG::2014-09-25 17:27:31,641::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70371::DEBUG::2014-09-25 17:27:31,645::task::595::Storage.TaskManager.Task::(_updateState) Task=`dcf712b2-c698-4080-903e-7518d6c3965d`::moving from state init -> state preparing >Thread-70371::INFO::2014-09-25 17:27:31,646::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70371::DEBUG::2014-09-25 17:27:31,648::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`925f435a-c5d1-4e5e-b9de-37deae0776d4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70371::DEBUG::2014-09-25 17:27:31,649::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70371::DEBUG::2014-09-25 17:27:31,650::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70371::DEBUG::2014-09-25 17:27:31,652::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`925f435a-c5d1-4e5e-b9de-37deae0776d4`::Granted request >Thread-70371::DEBUG::2014-09-25 17:27:31,654::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`dcf712b2-c698-4080-903e-7518d6c3965d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70371::DEBUG::2014-09-25 17:27:31,654::task::993::Storage.TaskManager.Task::(_decref) Task=`dcf712b2-c698-4080-903e-7518d6c3965d`::ref 1 aborting False >Thread-70371::INFO::2014-09-25 17:27:31,658::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70371::DEBUG::2014-09-25 17:27:31,659::task::1191::Storage.TaskManager.Task::(prepare) Task=`dcf712b2-c698-4080-903e-7518d6c3965d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70371::DEBUG::2014-09-25 17:27:31,659::task::595::Storage.TaskManager.Task::(_updateState) Task=`dcf712b2-c698-4080-903e-7518d6c3965d`::moving from state preparing -> state finished >Thread-70371::DEBUG::2014-09-25 17:27:31,659::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70371::DEBUG::2014-09-25 17:27:31,659::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70371::DEBUG::2014-09-25 17:27:31,660::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70371::DEBUG::2014-09-25 17:27:31,660::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70371::DEBUG::2014-09-25 17:27:31,660::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70371::DEBUG::2014-09-25 17:27:31,660::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70371::DEBUG::2014-09-25 17:27:31,661::task::993::Storage.TaskManager.Task::(_decref) Task=`dcf712b2-c698-4080-903e-7518d6c3965d`::ref 0 aborting False >Thread-70371::DEBUG::2014-09-25 17:27:31,661::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70371::DEBUG::2014-09-25 17:27:31,662::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:32,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:32,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020183 s, 50.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:32,696::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:32,698::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70372::DEBUG::2014-09-25 17:27:32,700::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:27:34,076::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:34,076::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:34,077::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:34,077::__init__::232::IOProcess::(_processLogs) (108258) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:34,078::__init__::232::IOProcess::(_processLogs) (108258) Queuing response >Thread-80::DEBUG::2014-09-25 17:27:34,079::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:27:34,088::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000448018 s, 844 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:34,089::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:34,089::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:34,090::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:34,091::__init__::232::IOProcess::(_processLogs) (108259) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:34,092::__init__::232::IOProcess::(_processLogs) (108259) Queuing response >Dummy-148::DEBUG::2014-09-25 17:27:34,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:34,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0280584 s, 36.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:34,746::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:34,747::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:34,747::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:34,747::__init__::232::IOProcess::(_processLogs) (108260) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:34,748::__init__::232::IOProcess::(_processLogs) (108260) Queuing response >Thread-79::DEBUG::2014-09-25 17:27:34,749::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:27:34,766::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00566123 s, 99.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:34,767::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:34,767::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:34,767::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:34,768::__init__::232::IOProcess::(_processLogs) (108261) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:34,769::__init__::232::IOProcess::(_processLogs) (108261) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:34,770::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:34,771::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:34,772::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:34,772::__init__::232::IOProcess::(_processLogs) (108262) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:34,772::__init__::232::IOProcess::(_processLogs) (108262) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:34,772::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:34,773::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:34,774::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:34,774::__init__::232::IOProcess::(_processLogs) (108263) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:34,774::__init__::232::IOProcess::(_processLogs) (108263) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:34,774::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:34,775::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:34,776::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:34,776::__init__::232::IOProcess::(_processLogs) (108264) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:34,776::__init__::232::IOProcess::(_processLogs) (108264) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:35,716::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:35,717::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70373::DEBUG::2014-09-25 17:27:35,719::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:36,397::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:36,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0252456 s, 40.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:27:38,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:38,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.219661 s, 4.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:38,736::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:38,738::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70374::DEBUG::2014-09-25 17:27:38,740::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:27:39,973::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:39,974::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:39,974::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:39,974::__init__::232::IOProcess::(_processLogs) (108265) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:39,980::__init__::232::IOProcess::(_processLogs) (108265) Queuing response >Thread-81::DEBUG::2014-09-25 17:27:39,982::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:27:40,007::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00879657 s, 42.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:40,008::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:40,008::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:40,009::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:40,009::__init__::232::IOProcess::(_processLogs) (108266) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:40,015::__init__::232::IOProcess::(_processLogs) (108266) Queuing response >Dummy-148::DEBUG::2014-09-25 17:27:40,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:40,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178679 s, 57.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:41,692::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:41,693::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70375::DEBUG::2014-09-25 17:27:41,693::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70375::DEBUG::2014-09-25 17:27:41,695::task::595::Storage.TaskManager.Task::(_updateState) Task=`5dc74165-fa03-4a73-b2f4-4b5b9e33695f`::moving from state init -> state preparing >Thread-70375::INFO::2014-09-25 17:27:41,695::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70375::INFO::2014-09-25 17:27:41,696::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70375::DEBUG::2014-09-25 17:27:41,696::task::1191::Storage.TaskManager.Task::(prepare) Task=`5dc74165-fa03-4a73-b2f4-4b5b9e33695f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70375::DEBUG::2014-09-25 17:27:41,696::task::595::Storage.TaskManager.Task::(_updateState) Task=`5dc74165-fa03-4a73-b2f4-4b5b9e33695f`::moving from state preparing -> state finished >Thread-70375::DEBUG::2014-09-25 17:27:41,697::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70375::DEBUG::2014-09-25 17:27:41,697::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70375::DEBUG::2014-09-25 17:27:41,697::task::993::Storage.TaskManager.Task::(_decref) Task=`5dc74165-fa03-4a73-b2f4-4b5b9e33695f`::ref 0 aborting False >Thread-70375::DEBUG::2014-09-25 17:27:41,697::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70375::DEBUG::2014-09-25 17:27:41,698::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:41,713::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:41,714::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70376::DEBUG::2014-09-25 17:27:41,714::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70376::DEBUG::2014-09-25 17:27:41,716::task::595::Storage.TaskManager.Task::(_updateState) Task=`4d2c0ca3-6529-4a86-8e81-111856e7d2fc`::moving from state init -> state preparing >Thread-70376::INFO::2014-09-25 17:27:41,716::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70376::DEBUG::2014-09-25 17:27:41,717::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8fd65e52-49d1-4f7c-a6e7-10c01cf50be9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70376::DEBUG::2014-09-25 17:27:41,717::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70376::DEBUG::2014-09-25 17:27:41,718::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70376::DEBUG::2014-09-25 17:27:41,718::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8fd65e52-49d1-4f7c-a6e7-10c01cf50be9`::Granted request >Thread-70376::DEBUG::2014-09-25 17:27:41,719::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4d2c0ca3-6529-4a86-8e81-111856e7d2fc`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70376::DEBUG::2014-09-25 17:27:41,719::task::993::Storage.TaskManager.Task::(_decref) Task=`4d2c0ca3-6529-4a86-8e81-111856e7d2fc`::ref 1 aborting False >Thread-70376::INFO::2014-09-25 17:27:41,723::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70376::DEBUG::2014-09-25 17:27:41,723::task::1191::Storage.TaskManager.Task::(prepare) Task=`4d2c0ca3-6529-4a86-8e81-111856e7d2fc`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70376::DEBUG::2014-09-25 17:27:41,724::task::595::Storage.TaskManager.Task::(_updateState) Task=`4d2c0ca3-6529-4a86-8e81-111856e7d2fc`::moving from state preparing -> state finished >Thread-70376::DEBUG::2014-09-25 17:27:41,724::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70376::DEBUG::2014-09-25 17:27:41,724::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70376::DEBUG::2014-09-25 17:27:41,725::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70376::DEBUG::2014-09-25 17:27:41,725::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70376::DEBUG::2014-09-25 17:27:41,725::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70376::DEBUG::2014-09-25 17:27:41,725::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70376::DEBUG::2014-09-25 17:27:41,726::task::993::Storage.TaskManager.Task::(_decref) Task=`4d2c0ca3-6529-4a86-8e81-111856e7d2fc`::ref 0 aborting False >Thread-70376::DEBUG::2014-09-25 17:27:41,726::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70376::DEBUG::2014-09-25 17:27:41,727::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:41,754::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:41,755::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70377::DEBUG::2014-09-25 17:27:41,757::task::595::Storage.TaskManager.Task::(_updateState) Task=`415c8b0f-7832-4961-a09a-a347ee0a5bb8`::moving from state init -> state preparing >Thread-70377::INFO::2014-09-25 17:27:41,758::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70377::INFO::2014-09-25 17:27:41,758::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00566123', 'lastCheck': '7.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000448018', 'lastCheck': '7.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00879657', 'lastCheck': '1.7', 'valid': True}} >Thread-70377::DEBUG::2014-09-25 17:27:41,758::task::1191::Storage.TaskManager.Task::(prepare) Task=`415c8b0f-7832-4961-a09a-a347ee0a5bb8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00566123', 'lastCheck': '7.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000448018', 'lastCheck': '7.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00879657', 'lastCheck': '1.7', 'valid': True}} >Thread-70377::DEBUG::2014-09-25 17:27:41,759::task::595::Storage.TaskManager.Task::(_updateState) Task=`415c8b0f-7832-4961-a09a-a347ee0a5bb8`::moving from state preparing -> state finished >Thread-70377::DEBUG::2014-09-25 17:27:41,759::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70377::DEBUG::2014-09-25 17:27:41,759::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70377::DEBUG::2014-09-25 17:27:41,759::task::993::Storage.TaskManager.Task::(_decref) Task=`415c8b0f-7832-4961-a09a-a347ee0a5bb8`::ref 0 aborting False >Thread-70377::DEBUG::2014-09-25 17:27:41,761::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:41,785::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:41,786::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70378::DEBUG::2014-09-25 17:27:41,788::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:42,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:42,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0206904 s, 49.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:44,092::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:44,093::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:44,093::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:44,093::__init__::232::IOProcess::(_processLogs) (108267) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:44,095::__init__::232::IOProcess::(_processLogs) (108267) Queuing response >Thread-80::DEBUG::2014-09-25 17:27:44,096::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:27:44,105::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000461865 s, 818 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:44,107::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:44,107::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:44,107::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:44,107::__init__::232::IOProcess::(_processLogs) (108268) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:44,108::__init__::232::IOProcess::(_processLogs) (108268) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:44,776::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:44,777::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:44,777::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:44,777::__init__::232::IOProcess::(_processLogs) (108269) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:44,781::__init__::232::IOProcess::(_processLogs) (108269) Queuing response >Thread-79::DEBUG::2014-09-25 17:27:44,782::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:27:44,799::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00872526 s, 64.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:44,801::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:44,801::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:44,802::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:44,802::__init__::232::IOProcess::(_processLogs) (108270) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:44,808::__init__::232::IOProcess::(_processLogs) (108270) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:44,809::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:44,810::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:44,810::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:44,810::__init__::232::IOProcess::(_processLogs) (108271) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:44,816::__init__::232::IOProcess::(_processLogs) (108271) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:44,817::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:44,817::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:44,818::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:44,818::__init__::232::IOProcess::(_processLogs) (108272) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:44,824::__init__::232::IOProcess::(_processLogs) (108272) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:44,825::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:44,825::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Dummy-148::DEBUG::2014-09-25 17:27:44,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:27:44,827::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:44,835::__init__::232::IOProcess::(_processLogs) (108273) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:44,836::__init__::232::IOProcess::(_processLogs) (108273) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:44,850::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:44,852::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70379::DEBUG::2014-09-25 17:27:44,854::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:44,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0197765 s, 51.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:27:46,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:46,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0276068 s, 37.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:47,871::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:47,872::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70380::DEBUG::2014-09-25 17:27:47,874::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:49,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:49,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166819 s, 61.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:50,017::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:50,018::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:50,018::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:50,018::__init__::232::IOProcess::(_processLogs) (108274) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:50,019::__init__::232::IOProcess::(_processLogs) (108274) Queuing response >Thread-81::DEBUG::2014-09-25 17:27:50,020::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:27:50,029::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000517723 s, 717 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:50,030::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:50,030::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:50,031::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:50,031::__init__::232::IOProcess::(_processLogs) (108275) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:50,031::__init__::232::IOProcess::(_processLogs) (108275) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:50,889::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:50,890::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70381::DEBUG::2014-09-25 17:27:50,892::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:51,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:51,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146865 s, 69.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:51,752::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70382::DEBUG::2014-09-25 17:27:51,753::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:27:51,753::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70382::DEBUG::2014-09-25 17:27:51,755::task::595::Storage.TaskManager.Task::(_updateState) Task=`f14614f6-aea1-4e91-bbcf-8e5ff87358ef`::moving from state init -> state preparing >Thread-70382::INFO::2014-09-25 17:27:51,756::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70382::INFO::2014-09-25 17:27:51,756::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70382::DEBUG::2014-09-25 17:27:51,757::task::1191::Storage.TaskManager.Task::(prepare) Task=`f14614f6-aea1-4e91-bbcf-8e5ff87358ef`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70382::DEBUG::2014-09-25 17:27:51,757::task::595::Storage.TaskManager.Task::(_updateState) Task=`f14614f6-aea1-4e91-bbcf-8e5ff87358ef`::moving from state preparing -> state finished >Thread-70382::DEBUG::2014-09-25 17:27:51,757::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70382::DEBUG::2014-09-25 17:27:51,757::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70382::DEBUG::2014-09-25 17:27:51,757::task::993::Storage.TaskManager.Task::(_decref) Task=`f14614f6-aea1-4e91-bbcf-8e5ff87358ef`::ref 0 aborting False >Thread-70382::DEBUG::2014-09-25 17:27:51,758::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70382::DEBUG::2014-09-25 17:27:51,758::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:51,769::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:51,770::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70383::DEBUG::2014-09-25 17:27:51,770::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70383::DEBUG::2014-09-25 17:27:51,772::task::595::Storage.TaskManager.Task::(_updateState) Task=`575c6b8b-3488-46af-8ab2-ecc2ba9ad6ca`::moving from state init -> state preparing >Thread-70383::INFO::2014-09-25 17:27:51,772::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70383::DEBUG::2014-09-25 17:27:51,773::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a33e96a7-7e56-4eb5-80f9-e1b312d8a1bf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70383::DEBUG::2014-09-25 17:27:51,773::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70383::DEBUG::2014-09-25 17:27:51,774::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70383::DEBUG::2014-09-25 17:27:51,774::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a33e96a7-7e56-4eb5-80f9-e1b312d8a1bf`::Granted request >Thread-70383::DEBUG::2014-09-25 17:27:51,774::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`575c6b8b-3488-46af-8ab2-ecc2ba9ad6ca`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70383::DEBUG::2014-09-25 17:27:51,775::task::993::Storage.TaskManager.Task::(_decref) Task=`575c6b8b-3488-46af-8ab2-ecc2ba9ad6ca`::ref 1 aborting False >Thread-70383::INFO::2014-09-25 17:27:51,779::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70383::DEBUG::2014-09-25 17:27:51,779::task::1191::Storage.TaskManager.Task::(prepare) Task=`575c6b8b-3488-46af-8ab2-ecc2ba9ad6ca`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70383::DEBUG::2014-09-25 17:27:51,780::task::595::Storage.TaskManager.Task::(_updateState) Task=`575c6b8b-3488-46af-8ab2-ecc2ba9ad6ca`::moving from state preparing -> state finished >Thread-70383::DEBUG::2014-09-25 17:27:51,780::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70383::DEBUG::2014-09-25 17:27:51,780::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70383::DEBUG::2014-09-25 17:27:51,780::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70383::DEBUG::2014-09-25 17:27:51,781::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70383::DEBUG::2014-09-25 17:27:51,781::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70383::DEBUG::2014-09-25 17:27:51,781::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70383::DEBUG::2014-09-25 17:27:51,782::task::993::Storage.TaskManager.Task::(_decref) Task=`575c6b8b-3488-46af-8ab2-ecc2ba9ad6ca`::ref 0 aborting False >Thread-70383::DEBUG::2014-09-25 17:27:51,782::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70383::DEBUG::2014-09-25 17:27:51,783::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:53,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:53,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163798 s, 62.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:53,905::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:53,907::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70384::DEBUG::2014-09-25 17:27:53,909::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:27:54,110::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:54,111::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:54,111::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:54,111::__init__::232::IOProcess::(_processLogs) (108276) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:54,111::__init__::232::IOProcess::(_processLogs) (108276) Queuing response >Thread-80::DEBUG::2014-09-25 17:27:54,113::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:27:54,122::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00047975 s, 788 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:54,123::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:54,123::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:54,123::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:54,123::__init__::232::IOProcess::(_processLogs) (108277) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:54,124::__init__::232::IOProcess::(_processLogs) (108277) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:54,837::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:54,837::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:54,838::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:54,838::__init__::232::IOProcess::(_processLogs) (108278) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:54,839::__init__::232::IOProcess::(_processLogs) (108278) Queuing response >Thread-79::DEBUG::2014-09-25 17:27:54,840::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:27:54,855::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00281637 s, 200 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:27:54,857::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:54,857::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:54,857::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:54,857::__init__::232::IOProcess::(_processLogs) (108279) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:27:54,861::__init__::232::IOProcess::(_processLogs) (108279) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:54,862::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:54,862::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:54,863::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:54,863::__init__::232::IOProcess::(_processLogs) (108280) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:54,863::__init__::232::IOProcess::(_processLogs) (108280) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:54,864::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:54,864::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:54,864::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:54,865::__init__::232::IOProcess::(_processLogs) (108281) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:54,865::__init__::232::IOProcess::(_processLogs) (108281) Queuing response >Thread-29::DEBUG::2014-09-25 17:27:54,866::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:27:54,866::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:27:54,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:27:54,867::__init__::232::IOProcess::(_processLogs) (108282) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:27:54,867::__init__::232::IOProcess::(_processLogs) (108282) Queuing response >Dummy-148::DEBUG::2014-09-25 17:27:55,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:55,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177129 s, 57.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:56,919::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:56,920::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70385::DEBUG::2014-09-25 17:27:56,923::task::595::Storage.TaskManager.Task::(_updateState) Task=`264964d2-fac7-44bb-914b-81647b68280c`::moving from state init -> state preparing >Thread-70385::INFO::2014-09-25 17:27:56,923::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70385::INFO::2014-09-25 17:27:56,924::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00281637', 'lastCheck': '2.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00047975', 'lastCheck': '2.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000517723', 'lastCheck': '6.9', 'valid': True}} >Thread-70385::DEBUG::2014-09-25 17:27:56,924::task::1191::Storage.TaskManager.Task::(prepare) Task=`264964d2-fac7-44bb-914b-81647b68280c`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00281637', 'lastCheck': '2.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00047975', 'lastCheck': '2.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000517723', 'lastCheck': '6.9', 'valid': True}} >Thread-70385::DEBUG::2014-09-25 17:27:56,924::task::595::Storage.TaskManager.Task::(_updateState) Task=`264964d2-fac7-44bb-914b-81647b68280c`::moving from state preparing -> state finished >Thread-70385::DEBUG::2014-09-25 17:27:56,925::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70385::DEBUG::2014-09-25 17:27:56,925::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70385::DEBUG::2014-09-25 17:27:56,925::task::993::Storage.TaskManager.Task::(_decref) Task=`264964d2-fac7-44bb-914b-81647b68280c`::ref 0 aborting False >Thread-70385::DEBUG::2014-09-25 17:27:56,927::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:27:56,941::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:27:56,942::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70386::DEBUG::2014-09-25 17:27:56,944::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:27:57,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:57,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135748 s, 75.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:27:59,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:27:59,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162234 s, 63.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:00,000::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:00,001::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70387::DEBUG::2014-09-25 17:28:00,003::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:28:00,034::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:00,034::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:00,034::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:00,034::__init__::232::IOProcess::(_processLogs) (108283) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:00,044::__init__::232::IOProcess::(_processLogs) (108283) Queuing response >Thread-81::DEBUG::2014-09-25 17:28:00,045::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:28:00,076::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00581422 s, 63.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:00,077::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:00,078::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:00,078::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:00,078::__init__::232::IOProcess::(_processLogs) (108284) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:00,084::__init__::232::IOProcess::(_processLogs) (108284) Queuing response >Dummy-148::DEBUG::2014-09-25 17:28:01,489::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:01,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173232 s, 59.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:01,806::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70388::DEBUG::2014-09-25 17:28:01,807::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:28:01,807::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70388::DEBUG::2014-09-25 17:28:01,809::task::595::Storage.TaskManager.Task::(_updateState) Task=`e2c169ed-b09d-45ed-bb05-b82a181a937e`::moving from state init -> state preparing >Thread-70388::INFO::2014-09-25 17:28:01,810::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70388::INFO::2014-09-25 17:28:01,810::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70388::DEBUG::2014-09-25 17:28:01,811::task::1191::Storage.TaskManager.Task::(prepare) Task=`e2c169ed-b09d-45ed-bb05-b82a181a937e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70388::DEBUG::2014-09-25 17:28:01,811::task::595::Storage.TaskManager.Task::(_updateState) Task=`e2c169ed-b09d-45ed-bb05-b82a181a937e`::moving from state preparing -> state finished >Thread-70388::DEBUG::2014-09-25 17:28:01,811::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70388::DEBUG::2014-09-25 17:28:01,811::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70388::DEBUG::2014-09-25 17:28:01,811::task::993::Storage.TaskManager.Task::(_decref) Task=`e2c169ed-b09d-45ed-bb05-b82a181a937e`::ref 0 aborting False >Thread-70388::DEBUG::2014-09-25 17:28:01,812::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70388::DEBUG::2014-09-25 17:28:01,812::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:01,823::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:01,825::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70389::DEBUG::2014-09-25 17:28:01,825::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70389::DEBUG::2014-09-25 17:28:01,826::task::595::Storage.TaskManager.Task::(_updateState) Task=`0e17a988-3f6a-4404-9179-d69bc5ed103e`::moving from state init -> state preparing >Thread-70389::INFO::2014-09-25 17:28:01,826::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70389::DEBUG::2014-09-25 17:28:01,827::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a8902254-bae7-4b6d-b873-7313b19d4ff5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70389::DEBUG::2014-09-25 17:28:01,827::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70389::DEBUG::2014-09-25 17:28:01,828::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70389::DEBUG::2014-09-25 17:28:01,828::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a8902254-bae7-4b6d-b873-7313b19d4ff5`::Granted request >Thread-70389::DEBUG::2014-09-25 17:28:01,829::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0e17a988-3f6a-4404-9179-d69bc5ed103e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70389::DEBUG::2014-09-25 17:28:01,829::task::993::Storage.TaskManager.Task::(_decref) Task=`0e17a988-3f6a-4404-9179-d69bc5ed103e`::ref 1 aborting False >Thread-70389::INFO::2014-09-25 17:28:01,833::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70389::DEBUG::2014-09-25 17:28:01,833::task::1191::Storage.TaskManager.Task::(prepare) Task=`0e17a988-3f6a-4404-9179-d69bc5ed103e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70389::DEBUG::2014-09-25 17:28:01,834::task::595::Storage.TaskManager.Task::(_updateState) Task=`0e17a988-3f6a-4404-9179-d69bc5ed103e`::moving from state preparing -> state finished >Thread-70389::DEBUG::2014-09-25 17:28:01,834::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70389::DEBUG::2014-09-25 17:28:01,834::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70389::DEBUG::2014-09-25 17:28:01,834::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70389::DEBUG::2014-09-25 17:28:01,835::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70389::DEBUG::2014-09-25 17:28:01,835::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70389::DEBUG::2014-09-25 17:28:01,835::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70389::DEBUG::2014-09-25 17:28:01,836::task::993::Storage.TaskManager.Task::(_decref) Task=`0e17a988-3f6a-4404-9179-d69bc5ed103e`::ref 0 aborting False >Thread-70389::DEBUG::2014-09-25 17:28:01,836::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70389::DEBUG::2014-09-25 17:28:01,837::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:03,017::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:03,018::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70390::DEBUG::2014-09-25 17:28:03,020::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:03,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:03,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0221647 s, 46.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:04,126::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:04,127::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:04,127::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:04,127::__init__::232::IOProcess::(_processLogs) (108285) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:04,128::__init__::232::IOProcess::(_processLogs) (108285) Queuing response >Thread-80::DEBUG::2014-09-25 17:28:04,129::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:28:04,138::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00045271 s, 835 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:04,139::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:04,140::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:04,140::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:04,140::__init__::232::IOProcess::(_processLogs) (108286) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:04,141::__init__::232::IOProcess::(_processLogs) (108286) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:04,868::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:04,868::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:04,869::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:04,869::__init__::232::IOProcess::(_processLogs) (108287) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:04,874::__init__::232::IOProcess::(_processLogs) (108287) Queuing response >Thread-79::DEBUG::2014-09-25 17:28:04,875::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:28:04,899::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00427894 s, 131 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:04,900::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:04,900::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:04,901::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:04,901::__init__::232::IOProcess::(_processLogs) (108288) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:04,904::__init__::232::IOProcess::(_processLogs) (108288) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:04,905::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:04,906::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:04,907::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:04,907::__init__::232::IOProcess::(_processLogs) (108289) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:04,907::__init__::232::IOProcess::(_processLogs) (108289) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:04,908::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:04,908::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:04,909::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:04,909::__init__::232::IOProcess::(_processLogs) (108290) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:04,909::__init__::232::IOProcess::(_processLogs) (108290) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:04,910::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:04,910::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:04,911::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:04,911::__init__::232::IOProcess::(_processLogs) (108291) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:04,912::__init__::232::IOProcess::(_processLogs) (108291) Queuing response >Dummy-148::DEBUG::2014-09-25 17:28:05,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:05,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164909 s, 62.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:06,031::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:06,033::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70391::DEBUG::2014-09-25 17:28:06,035::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:07,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:07,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191002 s, 53.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:09,052::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:09,054::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70392::DEBUG::2014-09-25 17:28:09,056::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:09,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:09,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013316 s, 76.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:10,086::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:10,087::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:10,087::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:10,087::__init__::232::IOProcess::(_processLogs) (108292) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:10,089::__init__::232::IOProcess::(_processLogs) (108292) Queuing response >Thread-81::DEBUG::2014-09-25 17:28:10,090::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:28:10,106::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00237817 s, 156 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:10,108::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:10,108::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:10,108::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:10,108::__init__::232::IOProcess::(_processLogs) (108293) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:10,111::__init__::232::IOProcess::(_processLogs) (108293) Queuing response >Dummy-148::DEBUG::2014-09-25 17:28:11,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:11,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0337751 s, 30.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:11,859::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70393::DEBUG::2014-09-25 17:28:11,861::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:28:11,861::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70393::DEBUG::2014-09-25 17:28:11,863::task::595::Storage.TaskManager.Task::(_updateState) Task=`179c040c-990c-408e-96c3-d75d618c8d04`::moving from state init -> state preparing >Thread-70393::INFO::2014-09-25 17:28:11,863::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70393::INFO::2014-09-25 17:28:11,864::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70393::DEBUG::2014-09-25 17:28:11,864::task::1191::Storage.TaskManager.Task::(prepare) Task=`179c040c-990c-408e-96c3-d75d618c8d04`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70393::DEBUG::2014-09-25 17:28:11,864::task::595::Storage.TaskManager.Task::(_updateState) Task=`179c040c-990c-408e-96c3-d75d618c8d04`::moving from state preparing -> state finished >Thread-70393::DEBUG::2014-09-25 17:28:11,865::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70393::DEBUG::2014-09-25 17:28:11,865::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70393::DEBUG::2014-09-25 17:28:11,865::task::993::Storage.TaskManager.Task::(_decref) Task=`179c040c-990c-408e-96c3-d75d618c8d04`::ref 0 aborting False >Thread-70393::DEBUG::2014-09-25 17:28:11,865::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70393::DEBUG::2014-09-25 17:28:11,866::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:11,876::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70394::DEBUG::2014-09-25 17:28:11,877::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:28:11,877::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70394::DEBUG::2014-09-25 17:28:11,879::task::595::Storage.TaskManager.Task::(_updateState) Task=`874f68a0-d12c-4915-b11c-18204c387508`::moving from state init -> state preparing >Thread-70394::INFO::2014-09-25 17:28:11,880::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70394::DEBUG::2014-09-25 17:28:11,880::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5ee5754a-b8e8-453f-a610-dd209bcbd462`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70394::DEBUG::2014-09-25 17:28:11,881::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70394::DEBUG::2014-09-25 17:28:11,881::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70394::DEBUG::2014-09-25 17:28:11,881::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5ee5754a-b8e8-453f-a610-dd209bcbd462`::Granted request >Thread-70394::DEBUG::2014-09-25 17:28:11,882::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`874f68a0-d12c-4915-b11c-18204c387508`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70394::DEBUG::2014-09-25 17:28:11,882::task::993::Storage.TaskManager.Task::(_decref) Task=`874f68a0-d12c-4915-b11c-18204c387508`::ref 1 aborting False >Thread-70394::INFO::2014-09-25 17:28:11,886::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70394::DEBUG::2014-09-25 17:28:11,887::task::1191::Storage.TaskManager.Task::(prepare) Task=`874f68a0-d12c-4915-b11c-18204c387508`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70394::DEBUG::2014-09-25 17:28:11,887::task::595::Storage.TaskManager.Task::(_updateState) Task=`874f68a0-d12c-4915-b11c-18204c387508`::moving from state preparing -> state finished >Thread-70394::DEBUG::2014-09-25 17:28:11,887::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70394::DEBUG::2014-09-25 17:28:11,887::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70394::DEBUG::2014-09-25 17:28:11,888::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70394::DEBUG::2014-09-25 17:28:11,888::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70394::DEBUG::2014-09-25 17:28:11,888::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70394::DEBUG::2014-09-25 17:28:11,889::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70394::DEBUG::2014-09-25 17:28:11,889::task::993::Storage.TaskManager.Task::(_decref) Task=`874f68a0-d12c-4915-b11c-18204c387508`::ref 0 aborting False >Thread-70394::DEBUG::2014-09-25 17:28:11,889::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70394::DEBUG::2014-09-25 17:28:11,890::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:12,071::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:12,072::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70395::DEBUG::2014-09-25 17:28:12,074::task::595::Storage.TaskManager.Task::(_updateState) Task=`10a17b48-7773-4840-b9b0-4d8c11ca25d4`::moving from state init -> state preparing >Thread-70395::INFO::2014-09-25 17:28:12,074::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70395::INFO::2014-09-25 17:28:12,075::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00427894', 'lastCheck': '7.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00045271', 'lastCheck': '7.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00237817', 'lastCheck': '2.0', 'valid': True}} >Thread-70395::DEBUG::2014-09-25 17:28:12,075::task::1191::Storage.TaskManager.Task::(prepare) Task=`10a17b48-7773-4840-b9b0-4d8c11ca25d4`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00427894', 'lastCheck': '7.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00045271', 'lastCheck': '7.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00237817', 'lastCheck': '2.0', 'valid': True}} >Thread-70395::DEBUG::2014-09-25 17:28:12,075::task::595::Storage.TaskManager.Task::(_updateState) Task=`10a17b48-7773-4840-b9b0-4d8c11ca25d4`::moving from state preparing -> state finished >Thread-70395::DEBUG::2014-09-25 17:28:12,075::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70395::DEBUG::2014-09-25 17:28:12,076::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70395::DEBUG::2014-09-25 17:28:12,076::task::993::Storage.TaskManager.Task::(_decref) Task=`10a17b48-7773-4840-b9b0-4d8c11ca25d4`::ref 0 aborting False >Thread-70395::DEBUG::2014-09-25 17:28:12,078::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:12,093::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:12,094::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70396::DEBUG::2014-09-25 17:28:12,096::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:13,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:13,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193821 s, 52.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:14,144::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:14,145::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:14,146::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:14,146::__init__::232::IOProcess::(_processLogs) (108294) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:14,147::__init__::232::IOProcess::(_processLogs) (108294) Queuing response >Thread-80::DEBUG::2014-09-25 17:28:14,148::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:28:14,157::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000546713 s, 691 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:14,158::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:14,158::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:14,158::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:14,159::__init__::232::IOProcess::(_processLogs) (108295) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:14,159::__init__::232::IOProcess::(_processLogs) (108295) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:14,912::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:14,912::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:14,912::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:14,913::__init__::232::IOProcess::(_processLogs) (108296) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:14,914::__init__::232::IOProcess::(_processLogs) (108296) Queuing response >Thread-79::DEBUG::2014-09-25 17:28:14,915::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:28:14,936::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.009115 s, 61.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:14,937::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:14,937::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:14,938::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:14,938::__init__::232::IOProcess::(_processLogs) (108297) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:14,941::__init__::232::IOProcess::(_processLogs) (108297) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:14,942::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:14,943::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:14,943::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:14,944::__init__::232::IOProcess::(_processLogs) (108298) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:14,944::__init__::232::IOProcess::(_processLogs) (108298) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:14,944::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:14,945::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:14,945::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:14,945::__init__::232::IOProcess::(_processLogs) (108299) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:14,946::__init__::232::IOProcess::(_processLogs) (108299) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:14,947::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:14,947::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:14,948::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:14,949::__init__::232::IOProcess::(_processLogs) (108300) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:14,949::__init__::232::IOProcess::(_processLogs) (108300) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:15,137::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:15,138::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70397::DEBUG::2014-09-25 17:28:15,141::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:15,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:15,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209147 s, 49.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:28:17,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:17,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0198651 s, 51.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:18,155::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:18,157::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70398::DEBUG::2014-09-25 17:28:18,160::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:19,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:19,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175041 s, 58.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:20,113::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:20,113::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:20,114::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:20,114::__init__::232::IOProcess::(_processLogs) (108301) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:20,116::__init__::232::IOProcess::(_processLogs) (108301) Queuing response >Thread-81::DEBUG::2014-09-25 17:28:20,117::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:28:20,134::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0041262 s, 89.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:20,135::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:20,136::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:20,136::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:20,136::__init__::232::IOProcess::(_processLogs) (108302) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:20,138::__init__::232::IOProcess::(_processLogs) (108302) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:21,175::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:21,176::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70399::DEBUG::2014-09-25 17:28:21,179::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:21,911::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:21,912::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70400::DEBUG::2014-09-25 17:28:21,912::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70400::DEBUG::2014-09-25 17:28:21,914::task::595::Storage.TaskManager.Task::(_updateState) Task=`bd81941a-fa59-4973-9017-4716c28da046`::moving from state init -> state preparing >Thread-70400::INFO::2014-09-25 17:28:21,914::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70400::INFO::2014-09-25 17:28:21,915::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70400::DEBUG::2014-09-25 17:28:21,915::task::1191::Storage.TaskManager.Task::(prepare) Task=`bd81941a-fa59-4973-9017-4716c28da046`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70400::DEBUG::2014-09-25 17:28:21,915::task::595::Storage.TaskManager.Task::(_updateState) Task=`bd81941a-fa59-4973-9017-4716c28da046`::moving from state preparing -> state finished >Thread-70400::DEBUG::2014-09-25 17:28:21,915::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70400::DEBUG::2014-09-25 17:28:21,916::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70400::DEBUG::2014-09-25 17:28:21,916::task::993::Storage.TaskManager.Task::(_decref) Task=`bd81941a-fa59-4973-9017-4716c28da046`::ref 0 aborting False >Thread-70400::DEBUG::2014-09-25 17:28:21,916::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70400::DEBUG::2014-09-25 17:28:21,917::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:21,927::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:21,929::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70401::DEBUG::2014-09-25 17:28:21,929::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70401::DEBUG::2014-09-25 17:28:21,931::task::595::Storage.TaskManager.Task::(_updateState) Task=`959b41ee-0389-4483-be24-449e19a21e1a`::moving from state init -> state preparing >Thread-70401::INFO::2014-09-25 17:28:21,931::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70401::DEBUG::2014-09-25 17:28:21,932::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9a7d1dd4-7efb-4ecf-a779-93718e7695d0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70401::DEBUG::2014-09-25 17:28:21,932::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70401::DEBUG::2014-09-25 17:28:21,932::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70401::DEBUG::2014-09-25 17:28:21,933::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9a7d1dd4-7efb-4ecf-a779-93718e7695d0`::Granted request >Thread-70401::DEBUG::2014-09-25 17:28:21,933::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`959b41ee-0389-4483-be24-449e19a21e1a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70401::DEBUG::2014-09-25 17:28:21,934::task::993::Storage.TaskManager.Task::(_decref) Task=`959b41ee-0389-4483-be24-449e19a21e1a`::ref 1 aborting False >Thread-70401::INFO::2014-09-25 17:28:21,938::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70401::DEBUG::2014-09-25 17:28:21,938::task::1191::Storage.TaskManager.Task::(prepare) Task=`959b41ee-0389-4483-be24-449e19a21e1a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70401::DEBUG::2014-09-25 17:28:21,938::task::595::Storage.TaskManager.Task::(_updateState) Task=`959b41ee-0389-4483-be24-449e19a21e1a`::moving from state preparing -> state finished >Thread-70401::DEBUG::2014-09-25 17:28:21,939::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70401::DEBUG::2014-09-25 17:28:21,939::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70401::DEBUG::2014-09-25 17:28:21,939::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70401::DEBUG::2014-09-25 17:28:21,939::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70401::DEBUG::2014-09-25 17:28:21,940::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70401::DEBUG::2014-09-25 17:28:21,940::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70401::DEBUG::2014-09-25 17:28:21,940::task::993::Storage.TaskManager.Task::(_decref) Task=`959b41ee-0389-4483-be24-449e19a21e1a`::ref 0 aborting False >Thread-70401::DEBUG::2014-09-25 17:28:21,941::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70401::DEBUG::2014-09-25 17:28:21,941::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:21,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:22,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129024 s, 79.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:28:24,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:24,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189815 s, 53.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:24,162::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:24,162::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:24,162::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:24,162::__init__::232::IOProcess::(_processLogs) (108303) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:24,163::__init__::232::IOProcess::(_processLogs) (108303) Queuing response >Thread-80::DEBUG::2014-09-25 17:28:24,164::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:28:24,173::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000487489 s, 775 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:24,174::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:24,174::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:24,175::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:24,175::__init__::232::IOProcess::(_processLogs) (108304) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:24,175::__init__::232::IOProcess::(_processLogs) (108304) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:24,195::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:24,196::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70402::DEBUG::2014-09-25 17:28:24,198::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:28:24,949::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:24,949::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:24,949::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:24,949::__init__::232::IOProcess::(_processLogs) (108305) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:24,951::__init__::232::IOProcess::(_processLogs) (108305) Queuing response >Thread-79::DEBUG::2014-09-25 17:28:24,952::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:28:24,966::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00181924 s, 309 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:24,968::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:24,968::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:24,968::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:24,969::__init__::232::IOProcess::(_processLogs) (108306) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:24,973::__init__::232::IOProcess::(_processLogs) (108306) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:24,974::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:24,974::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:24,975::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:24,975::__init__::232::IOProcess::(_processLogs) (108307) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:24,975::__init__::232::IOProcess::(_processLogs) (108307) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:24,976::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:24,977::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:24,977::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:24,977::__init__::232::IOProcess::(_processLogs) (108308) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:24,978::__init__::232::IOProcess::(_processLogs) (108308) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:24,978::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:24,979::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:24,979::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:24,980::__init__::232::IOProcess::(_processLogs) (108309) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:24,980::__init__::232::IOProcess::(_processLogs) (108309) Queuing response >Dummy-148::DEBUG::2014-09-25 17:28:26,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:26,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147883 s, 69.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:27,213::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:27,215::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70403::DEBUG::2014-09-25 17:28:27,218::task::595::Storage.TaskManager.Task::(_updateState) Task=`422b0aa8-dc6e-412e-ba0f-c6c427d27a55`::moving from state init -> state preparing >Thread-70403::INFO::2014-09-25 17:28:27,218::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70403::INFO::2014-09-25 17:28:27,218::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00181924', 'lastCheck': '2.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000487489', 'lastCheck': '3.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0041262', 'lastCheck': '7.1', 'valid': True}} >Thread-70403::DEBUG::2014-09-25 17:28:27,219::task::1191::Storage.TaskManager.Task::(prepare) Task=`422b0aa8-dc6e-412e-ba0f-c6c427d27a55`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00181924', 'lastCheck': '2.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000487489', 'lastCheck': '3.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0041262', 'lastCheck': '7.1', 'valid': True}} >Thread-70403::DEBUG::2014-09-25 17:28:27,219::task::595::Storage.TaskManager.Task::(_updateState) Task=`422b0aa8-dc6e-412e-ba0f-c6c427d27a55`::moving from state preparing -> state finished >Thread-70403::DEBUG::2014-09-25 17:28:27,219::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70403::DEBUG::2014-09-25 17:28:27,219::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70403::DEBUG::2014-09-25 17:28:27,220::task::993::Storage.TaskManager.Task::(_decref) Task=`422b0aa8-dc6e-412e-ba0f-c6c427d27a55`::ref 0 aborting False >Thread-70403::DEBUG::2014-09-25 17:28:27,222::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:27,237::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:27,238::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70404::DEBUG::2014-09-25 17:28:27,240::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:28,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:28,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186172 s, 55.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:30,141::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:30,142::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-81::DEBUG::2014-09-25 17:28:30,143::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:28:30,143::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:30,152::__init__::232::IOProcess::(_processLogs) (108310) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:30,152::__init__::232::IOProcess::(_processLogs) (108310) Queuing response >Thread-81::DEBUG::2014-09-25 17:28:30,153::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000386307 s, 960 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:30,155::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:30,155::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:30,155::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:30,156::__init__::232::IOProcess::(_processLogs) (108311) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:30,156::__init__::232::IOProcess::(_processLogs) (108311) Queuing response >Dummy-148::DEBUG::2014-09-25 17:28:30,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:30,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0127781 s, 80.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:30,284::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:30,286::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70405::DEBUG::2014-09-25 17:28:30,288::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:31,965::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:31,966::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70406::DEBUG::2014-09-25 17:28:31,966::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70406::DEBUG::2014-09-25 17:28:31,967::task::595::Storage.TaskManager.Task::(_updateState) Task=`b063c57f-2341-4c66-82b3-9e7d7870547b`::moving from state init -> state preparing >Thread-70406::INFO::2014-09-25 17:28:31,968::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70406::INFO::2014-09-25 17:28:31,968::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70406::DEBUG::2014-09-25 17:28:31,969::task::1191::Storage.TaskManager.Task::(prepare) Task=`b063c57f-2341-4c66-82b3-9e7d7870547b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70406::DEBUG::2014-09-25 17:28:31,969::task::595::Storage.TaskManager.Task::(_updateState) Task=`b063c57f-2341-4c66-82b3-9e7d7870547b`::moving from state preparing -> state finished >Thread-70406::DEBUG::2014-09-25 17:28:31,969::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70406::DEBUG::2014-09-25 17:28:31,969::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70406::DEBUG::2014-09-25 17:28:31,969::task::993::Storage.TaskManager.Task::(_decref) Task=`b063c57f-2341-4c66-82b3-9e7d7870547b`::ref 0 aborting False >Thread-70406::DEBUG::2014-09-25 17:28:31,970::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70406::DEBUG::2014-09-25 17:28:31,970::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:31,980::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:31,981::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70407::DEBUG::2014-09-25 17:28:31,981::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70407::DEBUG::2014-09-25 17:28:31,983::task::595::Storage.TaskManager.Task::(_updateState) Task=`f3d510ba-a19c-4f9b-8e99-21deb662b969`::moving from state init -> state preparing >Thread-70407::INFO::2014-09-25 17:28:31,983::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70407::DEBUG::2014-09-25 17:28:31,984::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`04ea357b-b428-46db-b8d5-4a90a7177e69`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70407::DEBUG::2014-09-25 17:28:31,984::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70407::DEBUG::2014-09-25 17:28:31,985::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70407::DEBUG::2014-09-25 17:28:31,985::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`04ea357b-b428-46db-b8d5-4a90a7177e69`::Granted request >Thread-70407::DEBUG::2014-09-25 17:28:31,986::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f3d510ba-a19c-4f9b-8e99-21deb662b969`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70407::DEBUG::2014-09-25 17:28:31,986::task::993::Storage.TaskManager.Task::(_decref) Task=`f3d510ba-a19c-4f9b-8e99-21deb662b969`::ref 1 aborting False >Thread-70407::INFO::2014-09-25 17:28:31,990::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70407::DEBUG::2014-09-25 17:28:31,990::task::1191::Storage.TaskManager.Task::(prepare) Task=`f3d510ba-a19c-4f9b-8e99-21deb662b969`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70407::DEBUG::2014-09-25 17:28:31,991::task::595::Storage.TaskManager.Task::(_updateState) Task=`f3d510ba-a19c-4f9b-8e99-21deb662b969`::moving from state preparing -> state finished >Thread-70407::DEBUG::2014-09-25 17:28:31,991::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70407::DEBUG::2014-09-25 17:28:31,991::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70407::DEBUG::2014-09-25 17:28:31,992::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70407::DEBUG::2014-09-25 17:28:31,992::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70407::DEBUG::2014-09-25 17:28:31,992::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70407::DEBUG::2014-09-25 17:28:31,992::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70407::DEBUG::2014-09-25 17:28:31,993::task::993::Storage.TaskManager.Task::(_decref) Task=`f3d510ba-a19c-4f9b-8e99-21deb662b969`::ref 0 aborting False >Thread-70407::DEBUG::2014-09-25 17:28:31,993::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70407::DEBUG::2014-09-25 17:28:31,994::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:32,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:32,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153541 s, 66.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:33,300::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:33,302::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70408::DEBUG::2014-09-25 17:28:33,304::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:28:34,178::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:34,178::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:34,178::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:34,179::__init__::232::IOProcess::(_processLogs) (108312) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:34,179::__init__::232::IOProcess::(_processLogs) (108312) Queuing response >Thread-80::DEBUG::2014-09-25 17:28:34,181::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:28:34,189::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000489785 s, 772 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:34,191::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:34,191::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:34,191::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:34,191::__init__::232::IOProcess::(_processLogs) (108313) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:34,192::__init__::232::IOProcess::(_processLogs) (108313) Queuing response >Dummy-148::DEBUG::2014-09-25 17:28:34,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:34,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142494 s, 71.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:34,980::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:34,981::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:34,981::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:34,981::__init__::232::IOProcess::(_processLogs) (108314) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:34,983::__init__::232::IOProcess::(_processLogs) (108314) Queuing response >Thread-79::DEBUG::2014-09-25 17:28:34,984::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:28:35,000::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00257566 s, 218 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:35,001::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:35,001::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:35,001::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:35,002::__init__::232::IOProcess::(_processLogs) (108315) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:35,004::__init__::232::IOProcess::(_processLogs) (108315) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:35,005::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:35,005::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:35,005::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:35,005::__init__::232::IOProcess::(_processLogs) (108316) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:35,006::__init__::232::IOProcess::(_processLogs) (108316) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:35,006::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:35,007::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:35,007::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:35,007::__init__::232::IOProcess::(_processLogs) (108317) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:35,007::__init__::232::IOProcess::(_processLogs) (108317) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:35,008::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:35,009::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:35,009::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:35,009::__init__::232::IOProcess::(_processLogs) (108318) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:35,009::__init__::232::IOProcess::(_processLogs) (108318) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:36,315::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:36,317::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70409::DEBUG::2014-09-25 17:28:36,318::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:36,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:36,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019234 s, 53.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:28:38,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:38,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188167 s, 54.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:39,340::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:39,342::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70410::DEBUG::2014-09-25 17:28:39,344::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:28:40,158::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:40,158::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:40,159::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:40,159::__init__::232::IOProcess::(_processLogs) (108319) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:40,161::__init__::232::IOProcess::(_processLogs) (108319) Queuing response >Thread-81::DEBUG::2014-09-25 17:28:40,162::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:28:40,179::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00798049 s, 46.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:40,180::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:40,181::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:40,181::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:40,181::__init__::232::IOProcess::(_processLogs) (108320) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:40,188::__init__::232::IOProcess::(_processLogs) (108320) Queuing response >Dummy-148::DEBUG::2014-09-25 17:28:40,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:40,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180937 s, 56.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:42,016::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70411::DEBUG::2014-09-25 17:28:42,017::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:28:42,017::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70411::DEBUG::2014-09-25 17:28:42,020::task::595::Storage.TaskManager.Task::(_updateState) Task=`c66974e8-2df4-4ffa-8a93-ca7e2f380d71`::moving from state init -> state preparing >Thread-70411::INFO::2014-09-25 17:28:42,020::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70411::INFO::2014-09-25 17:28:42,020::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70411::DEBUG::2014-09-25 17:28:42,021::task::1191::Storage.TaskManager.Task::(prepare) Task=`c66974e8-2df4-4ffa-8a93-ca7e2f380d71`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70411::DEBUG::2014-09-25 17:28:42,021::task::595::Storage.TaskManager.Task::(_updateState) Task=`c66974e8-2df4-4ffa-8a93-ca7e2f380d71`::moving from state preparing -> state finished >Thread-70411::DEBUG::2014-09-25 17:28:42,021::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70411::DEBUG::2014-09-25 17:28:42,021::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70411::DEBUG::2014-09-25 17:28:42,022::task::993::Storage.TaskManager.Task::(_decref) Task=`c66974e8-2df4-4ffa-8a93-ca7e2f380d71`::ref 0 aborting False >Thread-70411::DEBUG::2014-09-25 17:28:42,022::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70411::DEBUG::2014-09-25 17:28:42,022::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:42,033::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:42,034::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70412::DEBUG::2014-09-25 17:28:42,034::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70412::DEBUG::2014-09-25 17:28:42,036::task::595::Storage.TaskManager.Task::(_updateState) Task=`cfb12cc7-0821-4102-91ee-dacae1d9921a`::moving from state init -> state preparing >Thread-70412::INFO::2014-09-25 17:28:42,036::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70412::DEBUG::2014-09-25 17:28:42,037::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`84041ad5-0aa1-4e60-9835-1f0629957d1f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70412::DEBUG::2014-09-25 17:28:42,037::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70412::DEBUG::2014-09-25 17:28:42,038::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70412::DEBUG::2014-09-25 17:28:42,038::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`84041ad5-0aa1-4e60-9835-1f0629957d1f`::Granted request >Thread-70412::DEBUG::2014-09-25 17:28:42,038::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`cfb12cc7-0821-4102-91ee-dacae1d9921a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70412::DEBUG::2014-09-25 17:28:42,039::task::993::Storage.TaskManager.Task::(_decref) Task=`cfb12cc7-0821-4102-91ee-dacae1d9921a`::ref 1 aborting False >Thread-70412::INFO::2014-09-25 17:28:42,043::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70412::DEBUG::2014-09-25 17:28:42,043::task::1191::Storage.TaskManager.Task::(prepare) Task=`cfb12cc7-0821-4102-91ee-dacae1d9921a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70412::DEBUG::2014-09-25 17:28:42,044::task::595::Storage.TaskManager.Task::(_updateState) Task=`cfb12cc7-0821-4102-91ee-dacae1d9921a`::moving from state preparing -> state finished >Thread-70412::DEBUG::2014-09-25 17:28:42,044::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70412::DEBUG::2014-09-25 17:28:42,044::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70412::DEBUG::2014-09-25 17:28:42,044::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70412::DEBUG::2014-09-25 17:28:42,045::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70412::DEBUG::2014-09-25 17:28:42,045::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70412::DEBUG::2014-09-25 17:28:42,045::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70412::DEBUG::2014-09-25 17:28:42,045::task::993::Storage.TaskManager.Task::(_decref) Task=`cfb12cc7-0821-4102-91ee-dacae1d9921a`::ref 0 aborting False >Thread-70412::DEBUG::2014-09-25 17:28:42,046::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70412::DEBUG::2014-09-25 17:28:42,047::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:42,357::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:42,358::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70413::DEBUG::2014-09-25 17:28:42,360::task::595::Storage.TaskManager.Task::(_updateState) Task=`a215efcb-dbe0-45b7-b97d-315056c19194`::moving from state init -> state preparing >Thread-70413::INFO::2014-09-25 17:28:42,361::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70413::INFO::2014-09-25 17:28:42,361::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00257566', 'lastCheck': '7.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000489785', 'lastCheck': '8.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00798049', 'lastCheck': '2.2', 'valid': True}} >Thread-70413::DEBUG::2014-09-25 17:28:42,361::task::1191::Storage.TaskManager.Task::(prepare) Task=`a215efcb-dbe0-45b7-b97d-315056c19194`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00257566', 'lastCheck': '7.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000489785', 'lastCheck': '8.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00798049', 'lastCheck': '2.2', 'valid': True}} >Thread-70413::DEBUG::2014-09-25 17:28:42,362::task::595::Storage.TaskManager.Task::(_updateState) Task=`a215efcb-dbe0-45b7-b97d-315056c19194`::moving from state preparing -> state finished >Thread-70413::DEBUG::2014-09-25 17:28:42,362::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70413::DEBUG::2014-09-25 17:28:42,362::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70413::DEBUG::2014-09-25 17:28:42,362::task::993::Storage.TaskManager.Task::(_decref) Task=`a215efcb-dbe0-45b7-b97d-315056c19194`::ref 0 aborting False >Thread-70413::DEBUG::2014-09-25 17:28:42,364::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:42,387::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:42,388::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70414::DEBUG::2014-09-25 17:28:42,390::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:42,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:42,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200653 s, 51.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:44,194::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:44,195::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:44,195::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:44,195::__init__::232::IOProcess::(_processLogs) (108321) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:44,196::__init__::232::IOProcess::(_processLogs) (108321) Queuing response >Thread-80::DEBUG::2014-09-25 17:28:44,196::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:28:44,205::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000327574 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:44,206::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:44,207::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:44,207::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:44,209::__init__::232::IOProcess::(_processLogs) (108322) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:44,209::__init__::232::IOProcess::(_processLogs) (108322) Queuing response >Dummy-148::DEBUG::2014-09-25 17:28:44,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:44,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189071 s, 54.2 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 17:28:45,010::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:28:45,014::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,015::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,015::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,015::__init__::232::IOProcess::(_processLogs) (108323) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:28:45,016::__init__::232::IOProcess::(_processLogs) (108323) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:45,017::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,017::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,018::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,019::__init__::232::IOProcess::(_processLogs) (108324) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:28:45,019::__init__::232::IOProcess::(_processLogs) (108324) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:45,019::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,019::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,020::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,020::__init__::232::IOProcess::(_processLogs) (108325) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:28:45,021::__init__::232::IOProcess::(_processLogs) (108325) Queuing response >Thread-79::DEBUG::2014-09-25 17:28:45,025::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:28:45,026::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,026::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,026::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,027::__init__::232::IOProcess::(_processLogs) (108326) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:28:45,042::__init__::232::IOProcess::(_processLogs) (108326) Queuing response >Thread-79::DEBUG::2014-09-25 17:28:45,042::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:28:45,044::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,044::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,044::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,045::__init__::232::IOProcess::(_processLogs) (108327) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:28:45,045::__init__::232::IOProcess::(_processLogs) (108327) Queuing response >Thread-79::DEBUG::2014-09-25 17:28:45,046::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:28:45,047::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,047::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,048::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,049::__init__::232::IOProcess::(_processLogs) (108328) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:45,049::__init__::232::IOProcess::(_processLogs) (108328) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:45,049::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,050::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-79::DEBUG::2014-09-25 17:28:45,050::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:28:45,050::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::INFO::2014-09-25 17:28:45,051::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-29::DEBUG::2014-09-25 17:28:45,052::__init__::232::IOProcess::(_processLogs) (108329) Got request for method 'glob' >Thread-79::INFO::2014-09-25 17:28:45,052::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:28:45,053::__init__::232::IOProcess::(_processLogs) (108329) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:45,054::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,054::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,055::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,056::__init__::232::IOProcess::(_processLogs) (108330) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 17:28:45,056::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:28:45,057::__init__::232::IOProcess::(_processLogs) (108330) Queuing response >Thread-79::DEBUG::2014-09-25 17:28:45,066::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000284597 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:45,067::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,068::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,068::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,069::__init__::232::IOProcess::(_processLogs) (108331) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:45,070::__init__::232::IOProcess::(_processLogs) (108331) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:45,070::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,070::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,071::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,071::__init__::232::IOProcess::(_processLogs) (108332) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:45,071::__init__::232::IOProcess::(_processLogs) (108332) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:45,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,073::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,073::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,074::__init__::232::IOProcess::(_processLogs) (108333) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:45,074::__init__::232::IOProcess::(_processLogs) (108333) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:45,074::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:45,075::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:45,075::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:45,076::__init__::232::IOProcess::(_processLogs) (108334) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:45,076::__init__::232::IOProcess::(_processLogs) (108334) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:45,451::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:45,452::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70418::DEBUG::2014-09-25 17:28:45,455::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:46,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:46,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0295621 s, 34.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:48,466::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:48,468::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70419::DEBUG::2014-09-25 17:28:48,469::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:48,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:48,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0139997 s, 73.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:50,190::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:50,190::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:50,191::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:50,191::__init__::232::IOProcess::(_processLogs) (108335) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:50,194::__init__::232::IOProcess::(_processLogs) (108335) Queuing response >Thread-81::DEBUG::2014-09-25 17:28:50,195::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:28:50,210::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00336437 s, 110 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:50,211::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:50,211::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:50,211::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:50,212::__init__::232::IOProcess::(_processLogs) (108336) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:50,215::__init__::232::IOProcess::(_processLogs) (108336) Queuing response >Dummy-148::DEBUG::2014-09-25 17:28:50,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:50,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178036 s, 57.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:51,481::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:51,483::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70420::DEBUG::2014-09-25 17:28:51,485::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:52,069::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70421::DEBUG::2014-09-25 17:28:52,070::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:28:52,070::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70421::DEBUG::2014-09-25 17:28:52,072::task::595::Storage.TaskManager.Task::(_updateState) Task=`9e733da5-009c-4fe6-b5ff-9e8a67c766d6`::moving from state init -> state preparing >Thread-70421::INFO::2014-09-25 17:28:52,073::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70421::INFO::2014-09-25 17:28:52,073::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70421::DEBUG::2014-09-25 17:28:52,073::task::1191::Storage.TaskManager.Task::(prepare) Task=`9e733da5-009c-4fe6-b5ff-9e8a67c766d6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70421::DEBUG::2014-09-25 17:28:52,074::task::595::Storage.TaskManager.Task::(_updateState) Task=`9e733da5-009c-4fe6-b5ff-9e8a67c766d6`::moving from state preparing -> state finished >Thread-70421::DEBUG::2014-09-25 17:28:52,074::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70421::DEBUG::2014-09-25 17:28:52,074::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70421::DEBUG::2014-09-25 17:28:52,074::task::993::Storage.TaskManager.Task::(_decref) Task=`9e733da5-009c-4fe6-b5ff-9e8a67c766d6`::ref 0 aborting False >Thread-70421::DEBUG::2014-09-25 17:28:52,075::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70421::DEBUG::2014-09-25 17:28:52,075::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:52,085::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70422::DEBUG::2014-09-25 17:28:52,086::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:28:52,086::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70422::DEBUG::2014-09-25 17:28:52,088::task::595::Storage.TaskManager.Task::(_updateState) Task=`db9079f1-7305-40f4-8e8b-8def68015b8c`::moving from state init -> state preparing >Thread-70422::INFO::2014-09-25 17:28:52,089::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70422::DEBUG::2014-09-25 17:28:52,089::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f5ca87d2-d499-413c-86c3-a6a979a1a9c2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70422::DEBUG::2014-09-25 17:28:52,090::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70422::DEBUG::2014-09-25 17:28:52,090::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70422::DEBUG::2014-09-25 17:28:52,090::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f5ca87d2-d499-413c-86c3-a6a979a1a9c2`::Granted request >Thread-70422::DEBUG::2014-09-25 17:28:52,091::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`db9079f1-7305-40f4-8e8b-8def68015b8c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70422::DEBUG::2014-09-25 17:28:52,091::task::993::Storage.TaskManager.Task::(_decref) Task=`db9079f1-7305-40f4-8e8b-8def68015b8c`::ref 1 aborting False >Thread-70422::INFO::2014-09-25 17:28:52,096::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70422::DEBUG::2014-09-25 17:28:52,096::task::1191::Storage.TaskManager.Task::(prepare) Task=`db9079f1-7305-40f4-8e8b-8def68015b8c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70422::DEBUG::2014-09-25 17:28:52,096::task::595::Storage.TaskManager.Task::(_updateState) Task=`db9079f1-7305-40f4-8e8b-8def68015b8c`::moving from state preparing -> state finished >Thread-70422::DEBUG::2014-09-25 17:28:52,096::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70422::DEBUG::2014-09-25 17:28:52,097::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70422::DEBUG::2014-09-25 17:28:52,097::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70422::DEBUG::2014-09-25 17:28:52,097::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70422::DEBUG::2014-09-25 17:28:52,097::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70422::DEBUG::2014-09-25 17:28:52,098::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70422::DEBUG::2014-09-25 17:28:52,098::task::993::Storage.TaskManager.Task::(_decref) Task=`db9079f1-7305-40f4-8e8b-8def68015b8c`::ref 0 aborting False >Thread-70422::DEBUG::2014-09-25 17:28:52,098::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70422::DEBUG::2014-09-25 17:28:52,099::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:52,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:52,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157535 s, 65.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:54,210::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:54,210::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:54,211::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:54,211::__init__::232::IOProcess::(_processLogs) (108337) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:54,211::__init__::232::IOProcess::(_processLogs) (108337) Queuing response >Thread-80::DEBUG::2014-09-25 17:28:54,213::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:28:54,223::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000524314 s, 721 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:54,224::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:54,224::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:54,224::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:54,225::__init__::232::IOProcess::(_processLogs) (108338) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:54,225::__init__::232::IOProcess::(_processLogs) (108338) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:54,519::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:54,521::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70423::DEBUG::2014-09-25 17:28:54,523::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:54,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:54,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167782 s, 61.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:55,077::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:55,077::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:55,077::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:55,078::__init__::232::IOProcess::(_processLogs) (108339) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:55,082::__init__::232::IOProcess::(_processLogs) (108339) Queuing response >Thread-79::DEBUG::2014-09-25 17:28:55,083::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:28:55,096::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00139037 s, 404 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:28:55,097::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:55,097::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:55,097::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:55,098::__init__::232::IOProcess::(_processLogs) (108340) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:28:55,099::__init__::232::IOProcess::(_processLogs) (108340) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:55,101::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:55,101::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:55,101::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:55,102::__init__::232::IOProcess::(_processLogs) (108341) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:55,102::__init__::232::IOProcess::(_processLogs) (108341) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:55,103::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:55,103::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:55,104::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:55,104::__init__::232::IOProcess::(_processLogs) (108342) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:55,104::__init__::232::IOProcess::(_processLogs) (108342) Queuing response >Thread-29::DEBUG::2014-09-25 17:28:55,105::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:28:55,105::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:28:55,106::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:28:55,107::__init__::232::IOProcess::(_processLogs) (108343) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:28:55,107::__init__::232::IOProcess::(_processLogs) (108343) Queuing response >Dummy-148::DEBUG::2014-09-25 17:28:56,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:57,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164369 s, 62.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:57,536::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:57,538::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70424::DEBUG::2014-09-25 17:28:57,540::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6155593-7d33-4c60-b0b2-84c27e5b46e0`::moving from state init -> state preparing >Thread-70424::INFO::2014-09-25 17:28:57,541::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70424::INFO::2014-09-25 17:28:57,541::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00139037', 'lastCheck': '2.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000524314', 'lastCheck': '3.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00336437', 'lastCheck': '7.3', 'valid': True}} >Thread-70424::DEBUG::2014-09-25 17:28:57,542::task::1191::Storage.TaskManager.Task::(prepare) Task=`b6155593-7d33-4c60-b0b2-84c27e5b46e0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00139037', 'lastCheck': '2.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000524314', 'lastCheck': '3.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00336437', 'lastCheck': '7.3', 'valid': True}} >Thread-70424::DEBUG::2014-09-25 17:28:57,542::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6155593-7d33-4c60-b0b2-84c27e5b46e0`::moving from state preparing -> state finished >Thread-70424::DEBUG::2014-09-25 17:28:57,542::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70424::DEBUG::2014-09-25 17:28:57,542::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70424::DEBUG::2014-09-25 17:28:57,542::task::993::Storage.TaskManager.Task::(_decref) Task=`b6155593-7d33-4c60-b0b2-84c27e5b46e0`::ref 0 aborting False >Thread-70424::DEBUG::2014-09-25 17:28:57,545::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:28:57,565::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:28:57,566::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70425::DEBUG::2014-09-25 17:28:57,568::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:28:59,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:28:59,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211794 s, 48.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:00,216::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:00,217::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:00,217::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:00,217::__init__::232::IOProcess::(_processLogs) (108344) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:00,245::__init__::232::IOProcess::(_processLogs) (108344) Queuing response >Thread-81::DEBUG::2014-09-25 17:29:00,246::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:29:00,289::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00872668 s, 42.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:00,290::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:00,290::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:00,291::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:00,291::__init__::232::IOProcess::(_processLogs) (108345) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:00,297::__init__::232::IOProcess::(_processLogs) (108345) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:00,618::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:00,619::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70426::DEBUG::2014-09-25 17:29:00,621::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:01,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:01,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0217143 s, 47.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:02,121::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:02,122::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70427::DEBUG::2014-09-25 17:29:02,122::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70427::DEBUG::2014-09-25 17:29:02,124::task::595::Storage.TaskManager.Task::(_updateState) Task=`4db4944c-e8ae-4cb5-b8f9-985545ae0fb0`::moving from state init -> state preparing >Thread-70427::INFO::2014-09-25 17:29:02,124::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70427::INFO::2014-09-25 17:29:02,125::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70427::DEBUG::2014-09-25 17:29:02,125::task::1191::Storage.TaskManager.Task::(prepare) Task=`4db4944c-e8ae-4cb5-b8f9-985545ae0fb0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70427::DEBUG::2014-09-25 17:29:02,125::task::595::Storage.TaskManager.Task::(_updateState) Task=`4db4944c-e8ae-4cb5-b8f9-985545ae0fb0`::moving from state preparing -> state finished >Thread-70427::DEBUG::2014-09-25 17:29:02,126::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70427::DEBUG::2014-09-25 17:29:02,126::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70427::DEBUG::2014-09-25 17:29:02,126::task::993::Storage.TaskManager.Task::(_decref) Task=`4db4944c-e8ae-4cb5-b8f9-985545ae0fb0`::ref 0 aborting False >Thread-70427::DEBUG::2014-09-25 17:29:02,126::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70427::DEBUG::2014-09-25 17:29:02,127::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:02,137::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:02,139::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70428::DEBUG::2014-09-25 17:29:02,139::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70428::DEBUG::2014-09-25 17:29:02,140::task::595::Storage.TaskManager.Task::(_updateState) Task=`d167111a-7da5-42ba-8af3-b00318251e34`::moving from state init -> state preparing >Thread-70428::INFO::2014-09-25 17:29:02,142::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70428::DEBUG::2014-09-25 17:29:02,143::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`64ae4411-c5df-464a-83d5-eb5a92d653a0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70428::DEBUG::2014-09-25 17:29:02,143::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70428::DEBUG::2014-09-25 17:29:02,143::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70428::DEBUG::2014-09-25 17:29:02,143::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`64ae4411-c5df-464a-83d5-eb5a92d653a0`::Granted request >Thread-70428::DEBUG::2014-09-25 17:29:02,144::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`d167111a-7da5-42ba-8af3-b00318251e34`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70428::DEBUG::2014-09-25 17:29:02,144::task::993::Storage.TaskManager.Task::(_decref) Task=`d167111a-7da5-42ba-8af3-b00318251e34`::ref 1 aborting False >Thread-70428::INFO::2014-09-25 17:29:02,152::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70428::DEBUG::2014-09-25 17:29:02,152::task::1191::Storage.TaskManager.Task::(prepare) Task=`d167111a-7da5-42ba-8af3-b00318251e34`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70428::DEBUG::2014-09-25 17:29:02,152::task::595::Storage.TaskManager.Task::(_updateState) Task=`d167111a-7da5-42ba-8af3-b00318251e34`::moving from state preparing -> state finished >Thread-70428::DEBUG::2014-09-25 17:29:02,152::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70428::DEBUG::2014-09-25 17:29:02,153::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70428::DEBUG::2014-09-25 17:29:02,153::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70428::DEBUG::2014-09-25 17:29:02,153::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70428::DEBUG::2014-09-25 17:29:02,154::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70428::DEBUG::2014-09-25 17:29:02,154::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70428::DEBUG::2014-09-25 17:29:02,154::task::993::Storage.TaskManager.Task::(_decref) Task=`d167111a-7da5-42ba-8af3-b00318251e34`::ref 0 aborting False >Thread-70428::DEBUG::2014-09-25 17:29:02,155::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70428::DEBUG::2014-09-25 17:29:02,155::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:03,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:03,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135598 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:03,634::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:03,636::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70429::DEBUG::2014-09-25 17:29:03,639::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:29:04,228::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:04,228::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:04,228::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:04,229::__init__::232::IOProcess::(_processLogs) (108346) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:04,229::__init__::232::IOProcess::(_processLogs) (108346) Queuing response >Thread-80::DEBUG::2014-09-25 17:29:04,230::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:29:04,239::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000441472 s, 856 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:04,240::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:04,241::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:04,241::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:04,241::__init__::232::IOProcess::(_processLogs) (108347) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:04,242::__init__::232::IOProcess::(_processLogs) (108347) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:05,107::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:05,108::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:05,108::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:05,108::__init__::232::IOProcess::(_processLogs) (108348) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:05,115::__init__::232::IOProcess::(_processLogs) (108348) Queuing response >Thread-79::DEBUG::2014-09-25 17:29:05,116::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:29:05,133::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00278138 s, 202 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:05,134::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:05,134::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:05,135::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:05,135::__init__::232::IOProcess::(_processLogs) (108349) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:05,140::__init__::232::IOProcess::(_processLogs) (108349) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:05,142::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:05,143::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:05,144::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:05,144::__init__::232::IOProcess::(_processLogs) (108350) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:05,144::__init__::232::IOProcess::(_processLogs) (108350) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:05,145::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:05,145::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:05,145::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:05,145::__init__::232::IOProcess::(_processLogs) (108351) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:05,146::__init__::232::IOProcess::(_processLogs) (108351) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:05,147::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:05,147::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:05,148::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:05,150::__init__::232::IOProcess::(_processLogs) (108352) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:05,151::__init__::232::IOProcess::(_processLogs) (108352) Queuing response >Dummy-148::DEBUG::2014-09-25 17:29:05,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:05,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194954 s, 52.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:06,652::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:06,653::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70430::DEBUG::2014-09-25 17:29:06,655::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:07,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:07,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0220727 s, 46.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:29:09,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:09,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133023 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:09,670::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:09,672::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70431::DEBUG::2014-09-25 17:29:09,674::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:29:10,299::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:10,299::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:10,300::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:10,300::__init__::232::IOProcess::(_processLogs) (108353) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:10,304::__init__::232::IOProcess::(_processLogs) (108353) Queuing response >Thread-81::DEBUG::2014-09-25 17:29:10,305::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:29:10,320::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00212558 s, 175 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:10,321::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:10,321::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:10,322::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:10,322::__init__::232::IOProcess::(_processLogs) (108354) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:10,324::__init__::232::IOProcess::(_processLogs) (108354) Queuing response >Dummy-148::DEBUG::2014-09-25 17:29:11,392::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:11,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0197893 s, 51.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:12,184::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70432::DEBUG::2014-09-25 17:29:12,185::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:29:12,185::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70432::DEBUG::2014-09-25 17:29:12,188::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f2ad319-6e7c-4dd9-bd98-f1968bf785ab`::moving from state init -> state preparing >Thread-70432::INFO::2014-09-25 17:29:12,188::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70432::INFO::2014-09-25 17:29:12,189::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70432::DEBUG::2014-09-25 17:29:12,189::task::1191::Storage.TaskManager.Task::(prepare) Task=`0f2ad319-6e7c-4dd9-bd98-f1968bf785ab`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70432::DEBUG::2014-09-25 17:29:12,189::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f2ad319-6e7c-4dd9-bd98-f1968bf785ab`::moving from state preparing -> state finished >Thread-70432::DEBUG::2014-09-25 17:29:12,189::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70432::DEBUG::2014-09-25 17:29:12,190::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70432::DEBUG::2014-09-25 17:29:12,190::task::993::Storage.TaskManager.Task::(_decref) Task=`0f2ad319-6e7c-4dd9-bd98-f1968bf785ab`::ref 0 aborting False >Thread-70432::DEBUG::2014-09-25 17:29:12,190::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70432::DEBUG::2014-09-25 17:29:12,190::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:12,205::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:12,206::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70433::DEBUG::2014-09-25 17:29:12,206::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70433::DEBUG::2014-09-25 17:29:12,208::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b768f02-0258-433f-88ce-572602b09f16`::moving from state init -> state preparing >Thread-70433::INFO::2014-09-25 17:29:12,208::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70433::DEBUG::2014-09-25 17:29:12,209::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4caa6cd3-9f6d-4f89-875b-b9668de7891e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70433::DEBUG::2014-09-25 17:29:12,209::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70433::DEBUG::2014-09-25 17:29:12,209::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70433::DEBUG::2014-09-25 17:29:12,209::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4caa6cd3-9f6d-4f89-875b-b9668de7891e`::Granted request >Thread-70433::DEBUG::2014-09-25 17:29:12,210::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1b768f02-0258-433f-88ce-572602b09f16`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70433::DEBUG::2014-09-25 17:29:12,210::task::993::Storage.TaskManager.Task::(_decref) Task=`1b768f02-0258-433f-88ce-572602b09f16`::ref 1 aborting False >Thread-70433::INFO::2014-09-25 17:29:12,215::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70433::DEBUG::2014-09-25 17:29:12,215::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b768f02-0258-433f-88ce-572602b09f16`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70433::DEBUG::2014-09-25 17:29:12,215::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b768f02-0258-433f-88ce-572602b09f16`::moving from state preparing -> state finished >Thread-70433::DEBUG::2014-09-25 17:29:12,215::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70433::DEBUG::2014-09-25 17:29:12,216::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70433::DEBUG::2014-09-25 17:29:12,216::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70433::DEBUG::2014-09-25 17:29:12,216::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70433::DEBUG::2014-09-25 17:29:12,217::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70433::DEBUG::2014-09-25 17:29:12,217::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70433::DEBUG::2014-09-25 17:29:12,217::task::993::Storage.TaskManager.Task::(_decref) Task=`1b768f02-0258-433f-88ce-572602b09f16`::ref 0 aborting False >Thread-70433::DEBUG::2014-09-25 17:29:12,218::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70433::DEBUG::2014-09-25 17:29:12,218::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:12,687::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:12,688::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70434::DEBUG::2014-09-25 17:29:12,690::task::595::Storage.TaskManager.Task::(_updateState) Task=`14ac1d97-c5e6-4d52-b3fe-49d44e4b484e`::moving from state init -> state preparing >Thread-70434::INFO::2014-09-25 17:29:12,690::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70434::INFO::2014-09-25 17:29:12,691::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00278138', 'lastCheck': '7.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000441472', 'lastCheck': '8.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00212558', 'lastCheck': '2.4', 'valid': True}} >Thread-70434::DEBUG::2014-09-25 17:29:12,691::task::1191::Storage.TaskManager.Task::(prepare) Task=`14ac1d97-c5e6-4d52-b3fe-49d44e4b484e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00278138', 'lastCheck': '7.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000441472', 'lastCheck': '8.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00212558', 'lastCheck': '2.4', 'valid': True}} >Thread-70434::DEBUG::2014-09-25 17:29:12,691::task::595::Storage.TaskManager.Task::(_updateState) Task=`14ac1d97-c5e6-4d52-b3fe-49d44e4b484e`::moving from state preparing -> state finished >Thread-70434::DEBUG::2014-09-25 17:29:12,691::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70434::DEBUG::2014-09-25 17:29:12,692::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70434::DEBUG::2014-09-25 17:29:12,692::task::993::Storage.TaskManager.Task::(_decref) Task=`14ac1d97-c5e6-4d52-b3fe-49d44e4b484e`::ref 0 aborting False >Thread-70434::DEBUG::2014-09-25 17:29:12,694::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:12,714::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:12,715::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70435::DEBUG::2014-09-25 17:29:12,717::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:13,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:13,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156322 s, 65.5 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 17:29:14,243::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:29:14,247::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:14,248::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:14,248::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:14,249::__init__::232::IOProcess::(_processLogs) (108355) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:29:14,249::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:14,249::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:14,250::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:14,250::__init__::232::IOProcess::(_processLogs) (108356) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:29:14,250::__init__::232::IOProcess::(_processLogs) (108356) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:14,251::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:14,252::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:14,253::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:14,255::__init__::232::IOProcess::(_processLogs) (108357) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:29:14,258::__init__::232::IOProcess::(_processLogs) (108355) Queuing response >Thread-80::DEBUG::2014-09-25 17:29:14,261::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:29:14,262::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:14,262::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:14,262::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:14,263::__init__::232::IOProcess::(_processLogs) (108358) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:29:14,263::__init__::232::IOProcess::(_processLogs) (108357) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:14,266::__init__::232::IOProcess::(_processLogs) (108358) Queuing response >Thread-80::DEBUG::2014-09-25 17:29:14,266::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:29:14,267::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:14,267::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:14,268::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:14,268::__init__::232::IOProcess::(_processLogs) (108359) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:29:14,268::__init__::232::IOProcess::(_processLogs) (108359) Queuing response >Thread-80::DEBUG::2014-09-25 17:29:14,269::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:29:14,271::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:14,271::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:14,272::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:14,272::__init__::232::IOProcess::(_processLogs) (108360) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:14,272::__init__::232::IOProcess::(_processLogs) (108360) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:14,273::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:14,273::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:14,273::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:14,274::__init__::232::IOProcess::(_processLogs) (108361) Got request for method 'glob' >Thread-80::DEBUG::2014-09-25 17:29:14,274::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:29:14,274::__init__::232::IOProcess::(_processLogs) (108361) Queuing response >Thread-80::INFO::2014-09-25 17:29:14,275::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:29:14,276::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:29:14,277::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:14,277::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:14,278::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:29:14,279::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:29:14,279::__init__::232::IOProcess::(_processLogs) (108362) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:14,287::__init__::232::IOProcess::(_processLogs) (108362) Queuing response >Thread-80::DEBUG::2014-09-25 17:29:14,288::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000444068 s, 851 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:14,289::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:14,290::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:14,290::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:14,290::__init__::232::IOProcess::(_processLogs) (108363) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:14,291::__init__::232::IOProcess::(_processLogs) (108363) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:15,150::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:15,150::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:15,150::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:15,151::__init__::232::IOProcess::(_processLogs) (108364) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:15,161::__init__::232::IOProcess::(_processLogs) (108364) Queuing response >Thread-79::DEBUG::2014-09-25 17:29:15,162::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:29:15,182::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0036728 s, 153 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:15,184::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:15,184::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:15,184::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:15,185::__init__::232::IOProcess::(_processLogs) (108365) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:15,186::__init__::232::IOProcess::(_processLogs) (108365) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:15,188::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:15,188::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:15,188::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:15,188::__init__::232::IOProcess::(_processLogs) (108366) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:15,189::__init__::232::IOProcess::(_processLogs) (108366) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:15,190::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:15,190::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:15,190::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:15,190::__init__::232::IOProcess::(_processLogs) (108367) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:15,191::__init__::232::IOProcess::(_processLogs) (108367) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:15,191::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:15,192::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:15,192::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:15,193::__init__::232::IOProcess::(_processLogs) (108368) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:15,193::__init__::232::IOProcess::(_processLogs) (108368) Queuing response >Dummy-148::DEBUG::2014-09-25 17:29:15,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:15,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135962 s, 75.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:15,764::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:15,766::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70439::DEBUG::2014-09-25 17:29:15,768::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:17,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:17,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211501 s, 48.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:18,780::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:18,781::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70440::DEBUG::2014-09-25 17:29:18,783::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:19,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:19,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209039 s, 49.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:20,326::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:20,326::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:20,326::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:20,327::__init__::232::IOProcess::(_processLogs) (108369) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:20,327::__init__::232::IOProcess::(_processLogs) (108369) Queuing response >Thread-81::DEBUG::2014-09-25 17:29:20,328::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:29:20,345::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00520607 s, 71.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:20,346::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:20,347::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:20,347::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:20,347::__init__::232::IOProcess::(_processLogs) (108370) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:20,349::__init__::232::IOProcess::(_processLogs) (108370) Queuing response >Dummy-148::DEBUG::2014-09-25 17:29:21,645::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:21,677::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218883 s, 46.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:21,794::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:21,795::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70441::DEBUG::2014-09-25 17:29:21,797::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:22,241::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70442::DEBUG::2014-09-25 17:29:22,242::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:29:22,242::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70442::DEBUG::2014-09-25 17:29:22,245::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf18f878-00e0-4665-b6e4-9b9a41d2aa75`::moving from state init -> state preparing >Thread-70442::INFO::2014-09-25 17:29:22,245::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70442::INFO::2014-09-25 17:29:22,246::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70442::DEBUG::2014-09-25 17:29:22,246::task::1191::Storage.TaskManager.Task::(prepare) Task=`bf18f878-00e0-4665-b6e4-9b9a41d2aa75`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70442::DEBUG::2014-09-25 17:29:22,246::task::595::Storage.TaskManager.Task::(_updateState) Task=`bf18f878-00e0-4665-b6e4-9b9a41d2aa75`::moving from state preparing -> state finished >Thread-70442::DEBUG::2014-09-25 17:29:22,246::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70442::DEBUG::2014-09-25 17:29:22,247::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70442::DEBUG::2014-09-25 17:29:22,247::task::993::Storage.TaskManager.Task::(_decref) Task=`bf18f878-00e0-4665-b6e4-9b9a41d2aa75`::ref 0 aborting False >Thread-70442::DEBUG::2014-09-25 17:29:22,247::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70442::DEBUG::2014-09-25 17:29:22,247::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:22,257::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70443::DEBUG::2014-09-25 17:29:22,258::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:29:22,259::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70443::DEBUG::2014-09-25 17:29:22,261::task::595::Storage.TaskManager.Task::(_updateState) Task=`3396e617-d60e-4bee-b10e-de81a179e337`::moving from state init -> state preparing >Thread-70443::INFO::2014-09-25 17:29:22,261::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70443::DEBUG::2014-09-25 17:29:22,262::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5de33f82-1a2e-4e20-9093-0f5f08663ee1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70443::DEBUG::2014-09-25 17:29:22,262::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70443::DEBUG::2014-09-25 17:29:22,262::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70443::DEBUG::2014-09-25 17:29:22,262::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5de33f82-1a2e-4e20-9093-0f5f08663ee1`::Granted request >Thread-70443::DEBUG::2014-09-25 17:29:22,263::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3396e617-d60e-4bee-b10e-de81a179e337`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70443::DEBUG::2014-09-25 17:29:22,263::task::993::Storage.TaskManager.Task::(_decref) Task=`3396e617-d60e-4bee-b10e-de81a179e337`::ref 1 aborting False >Thread-70443::INFO::2014-09-25 17:29:22,268::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70443::DEBUG::2014-09-25 17:29:22,268::task::1191::Storage.TaskManager.Task::(prepare) Task=`3396e617-d60e-4bee-b10e-de81a179e337`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70443::DEBUG::2014-09-25 17:29:22,268::task::595::Storage.TaskManager.Task::(_updateState) Task=`3396e617-d60e-4bee-b10e-de81a179e337`::moving from state preparing -> state finished >Thread-70443::DEBUG::2014-09-25 17:29:22,268::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70443::DEBUG::2014-09-25 17:29:22,269::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70443::DEBUG::2014-09-25 17:29:22,269::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70443::DEBUG::2014-09-25 17:29:22,269::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70443::DEBUG::2014-09-25 17:29:22,269::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70443::DEBUG::2014-09-25 17:29:22,270::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70443::DEBUG::2014-09-25 17:29:22,270::task::993::Storage.TaskManager.Task::(_decref) Task=`3396e617-d60e-4bee-b10e-de81a179e337`::ref 0 aborting False >Thread-70443::DEBUG::2014-09-25 17:29:22,271::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70443::DEBUG::2014-09-25 17:29:22,271::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:23,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:23,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201659 s, 50.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:24,293::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:24,293::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:24,294::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:24,294::__init__::232::IOProcess::(_processLogs) (108371) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:24,294::__init__::232::IOProcess::(_processLogs) (108371) Queuing response >Thread-80::DEBUG::2014-09-25 17:29:24,296::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:29:24,304::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000483772 s, 781 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:24,305::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:24,305::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:24,306::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:24,306::__init__::232::IOProcess::(_processLogs) (108372) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:24,306::__init__::232::IOProcess::(_processLogs) (108372) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:24,812::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:24,813::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70444::DEBUG::2014-09-25 17:29:24,815::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:29:25,194::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:25,194::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:25,194::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:25,194::__init__::232::IOProcess::(_processLogs) (108373) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:25,197::__init__::232::IOProcess::(_processLogs) (108373) Queuing response >Thread-79::DEBUG::2014-09-25 17:29:25,198::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:29:25,206::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000362208 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:25,207::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:25,208::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:25,208::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:25,208::__init__::232::IOProcess::(_processLogs) (108374) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:25,209::__init__::232::IOProcess::(_processLogs) (108374) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:25,210::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:25,211::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:25,211::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:25,211::__init__::232::IOProcess::(_processLogs) (108375) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:25,212::__init__::232::IOProcess::(_processLogs) (108375) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:25,212::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:25,213::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:25,213::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:25,213::__init__::232::IOProcess::(_processLogs) (108376) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:25,214::__init__::232::IOProcess::(_processLogs) (108376) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:25,214::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:25,215::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:25,215::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:25,216::__init__::232::IOProcess::(_processLogs) (108377) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:25,216::__init__::232::IOProcess::(_processLogs) (108377) Queuing response >Dummy-148::DEBUG::2014-09-25 17:29:25,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:25,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020369 s, 50.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:29:27,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:27,830::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:27,833::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70445::DEBUG::2014-09-25 17:29:27,842::task::595::Storage.TaskManager.Task::(_updateState) Task=`62dae087-ff86-4c86-a690-07448e89d945`::moving from state init -> state preparing >Thread-70445::INFO::2014-09-25 17:29:27,843::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70445::INFO::2014-09-25 17:29:27,846::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000362208', 'lastCheck': '2.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000483772', 'lastCheck': '3.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00520607', 'lastCheck': '7.5', 'valid': True}} >Thread-70445::DEBUG::2014-09-25 17:29:27,847::task::1191::Storage.TaskManager.Task::(prepare) Task=`62dae087-ff86-4c86-a690-07448e89d945`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000362208', 'lastCheck': '2.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000483772', 'lastCheck': '3.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00520607', 'lastCheck': '7.5', 'valid': True}} >Thread-70445::DEBUG::2014-09-25 17:29:27,848::task::595::Storage.TaskManager.Task::(_updateState) Task=`62dae087-ff86-4c86-a690-07448e89d945`::moving from state preparing -> state finished >Thread-70445::DEBUG::2014-09-25 17:29:27,849::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70445::DEBUG::2014-09-25 17:29:27,850::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Dummy-148::DEBUG::2014-09-25 17:29:27,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0279265 s, 36.7 MB/s\n'; <rc> = 0 >Thread-70445::DEBUG::2014-09-25 17:29:27,852::task::993::Storage.TaskManager.Task::(_decref) Task=`62dae087-ff86-4c86-a690-07448e89d945`::ref 0 aborting False >Thread-70445::DEBUG::2014-09-25 17:29:27,860::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:27,879::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:27,882::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70446::DEBUG::2014-09-25 17:29:27,884::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:29,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:29,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201732 s, 50.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:30,350::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:30,351::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:30,351::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:30,351::__init__::232::IOProcess::(_processLogs) (108378) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:30,352::__init__::232::IOProcess::(_processLogs) (108378) Queuing response >Thread-81::DEBUG::2014-09-25 17:29:30,353::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:29:30,369::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0031194 s, 119 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:30,370::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:30,370::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:30,371::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:30,371::__init__::232::IOProcess::(_processLogs) (108379) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:30,374::__init__::232::IOProcess::(_processLogs) (108379) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:30,934::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:30,936::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70447::DEBUG::2014-09-25 17:29:30,938::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:31,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:31,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014137 s, 72.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:32,294::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:32,296::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70448::DEBUG::2014-09-25 17:29:32,296::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70448::DEBUG::2014-09-25 17:29:32,298::task::595::Storage.TaskManager.Task::(_updateState) Task=`51b8ff65-99d9-4c77-9653-0f8a22731a0f`::moving from state init -> state preparing >Thread-70448::INFO::2014-09-25 17:29:32,298::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70448::INFO::2014-09-25 17:29:32,298::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70448::DEBUG::2014-09-25 17:29:32,299::task::1191::Storage.TaskManager.Task::(prepare) Task=`51b8ff65-99d9-4c77-9653-0f8a22731a0f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70448::DEBUG::2014-09-25 17:29:32,299::task::595::Storage.TaskManager.Task::(_updateState) Task=`51b8ff65-99d9-4c77-9653-0f8a22731a0f`::moving from state preparing -> state finished >Thread-70448::DEBUG::2014-09-25 17:29:32,299::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70448::DEBUG::2014-09-25 17:29:32,299::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70448::DEBUG::2014-09-25 17:29:32,300::task::993::Storage.TaskManager.Task::(_decref) Task=`51b8ff65-99d9-4c77-9653-0f8a22731a0f`::ref 0 aborting False >Thread-70448::DEBUG::2014-09-25 17:29:32,300::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70448::DEBUG::2014-09-25 17:29:32,300::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:32,311::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:32,312::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70449::DEBUG::2014-09-25 17:29:32,312::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70449::DEBUG::2014-09-25 17:29:32,314::task::595::Storage.TaskManager.Task::(_updateState) Task=`a717f65e-5b71-4758-8bdf-731d683c9000`::moving from state init -> state preparing >Thread-70449::INFO::2014-09-25 17:29:32,314::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70449::DEBUG::2014-09-25 17:29:32,315::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a6b7eda7-b74f-409c-b65d-92beb49bc278`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70449::DEBUG::2014-09-25 17:29:32,315::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70449::DEBUG::2014-09-25 17:29:32,315::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70449::DEBUG::2014-09-25 17:29:32,316::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a6b7eda7-b74f-409c-b65d-92beb49bc278`::Granted request >Thread-70449::DEBUG::2014-09-25 17:29:32,316::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a717f65e-5b71-4758-8bdf-731d683c9000`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70449::DEBUG::2014-09-25 17:29:32,316::task::993::Storage.TaskManager.Task::(_decref) Task=`a717f65e-5b71-4758-8bdf-731d683c9000`::ref 1 aborting False >Thread-70449::INFO::2014-09-25 17:29:32,321::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70449::DEBUG::2014-09-25 17:29:32,321::task::1191::Storage.TaskManager.Task::(prepare) Task=`a717f65e-5b71-4758-8bdf-731d683c9000`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70449::DEBUG::2014-09-25 17:29:32,321::task::595::Storage.TaskManager.Task::(_updateState) Task=`a717f65e-5b71-4758-8bdf-731d683c9000`::moving from state preparing -> state finished >Thread-70449::DEBUG::2014-09-25 17:29:32,321::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70449::DEBUG::2014-09-25 17:29:32,322::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70449::DEBUG::2014-09-25 17:29:32,322::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70449::DEBUG::2014-09-25 17:29:32,322::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70449::DEBUG::2014-09-25 17:29:32,323::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70449::DEBUG::2014-09-25 17:29:32,323::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70449::DEBUG::2014-09-25 17:29:32,323::task::993::Storage.TaskManager.Task::(_decref) Task=`a717f65e-5b71-4758-8bdf-731d683c9000`::ref 0 aborting False >Thread-70449::DEBUG::2014-09-25 17:29:32,324::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70449::DEBUG::2014-09-25 17:29:32,324::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:33,953::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:33,954::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70450::DEBUG::2014-09-25 17:29:33,956::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:33,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:34,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155265 s, 66.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:34,309::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:34,309::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:34,309::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:34,310::__init__::232::IOProcess::(_processLogs) (108380) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:34,310::__init__::232::IOProcess::(_processLogs) (108380) Queuing response >Thread-80::DEBUG::2014-09-25 17:29:34,311::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:29:34,320::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000426093 s, 887 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:34,321::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:34,321::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:34,322::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:34,322::__init__::232::IOProcess::(_processLogs) (108381) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:34,322::__init__::232::IOProcess::(_processLogs) (108381) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:35,216::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:35,217::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:35,217::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:35,217::__init__::232::IOProcess::(_processLogs) (108382) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:35,221::__init__::232::IOProcess::(_processLogs) (108382) Queuing response >Thread-79::DEBUG::2014-09-25 17:29:35,222::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:29:35,235::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0028078 s, 200 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:35,236::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:35,236::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:35,236::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:35,237::__init__::232::IOProcess::(_processLogs) (108383) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:35,244::__init__::232::IOProcess::(_processLogs) (108383) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:35,245::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:35,246::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:35,246::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:35,247::__init__::232::IOProcess::(_processLogs) (108384) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:35,247::__init__::232::IOProcess::(_processLogs) (108384) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:35,247::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:35,248::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:35,248::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:35,248::__init__::232::IOProcess::(_processLogs) (108385) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:35,248::__init__::232::IOProcess::(_processLogs) (108385) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:35,249::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:35,249::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:35,250::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:35,250::__init__::232::IOProcess::(_processLogs) (108386) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:35,250::__init__::232::IOProcess::(_processLogs) (108386) Queuing response >Dummy-148::DEBUG::2014-09-25 17:29:36,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:36,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211666 s, 48.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:36,971::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:36,973::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70451::DEBUG::2014-09-25 17:29:36,975::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:38,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:38,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155123 s, 66.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:39,989::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:39,990::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70452::DEBUG::2014-09-25 17:29:39,993::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:40,130::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:40,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166086 s, 61.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:40,376::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:40,376::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:40,377::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:40,377::__init__::232::IOProcess::(_processLogs) (108387) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:40,379::__init__::232::IOProcess::(_processLogs) (108387) Queuing response >Thread-81::DEBUG::2014-09-25 17:29:40,380::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:29:40,394::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00250088 s, 148 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:40,395::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:40,395::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:40,396::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:40,396::__init__::232::IOProcess::(_processLogs) (108388) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:40,396::__init__::232::IOProcess::(_processLogs) (108388) Queuing response >Dummy-148::DEBUG::2014-09-25 17:29:42,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:42,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214728 s, 47.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:42,347::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70453::DEBUG::2014-09-25 17:29:42,348::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:29:42,349::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70453::DEBUG::2014-09-25 17:29:42,351::task::595::Storage.TaskManager.Task::(_updateState) Task=`d3b89e61-c127-4c76-a8fe-57d759aa4ed3`::moving from state init -> state preparing >Thread-70453::INFO::2014-09-25 17:29:42,351::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70453::INFO::2014-09-25 17:29:42,352::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70453::DEBUG::2014-09-25 17:29:42,352::task::1191::Storage.TaskManager.Task::(prepare) Task=`d3b89e61-c127-4c76-a8fe-57d759aa4ed3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70453::DEBUG::2014-09-25 17:29:42,353::task::595::Storage.TaskManager.Task::(_updateState) Task=`d3b89e61-c127-4c76-a8fe-57d759aa4ed3`::moving from state preparing -> state finished >Thread-70453::DEBUG::2014-09-25 17:29:42,353::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70453::DEBUG::2014-09-25 17:29:42,353::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70453::DEBUG::2014-09-25 17:29:42,353::task::993::Storage.TaskManager.Task::(_decref) Task=`d3b89e61-c127-4c76-a8fe-57d759aa4ed3`::ref 0 aborting False >Thread-70453::DEBUG::2014-09-25 17:29:42,354::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70453::DEBUG::2014-09-25 17:29:42,354::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:42,364::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:42,365::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70454::DEBUG::2014-09-25 17:29:42,365::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70454::DEBUG::2014-09-25 17:29:42,367::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2cd6203-91fd-453d-add6-cc439802811d`::moving from state init -> state preparing >Thread-70454::INFO::2014-09-25 17:29:42,367::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70454::DEBUG::2014-09-25 17:29:42,368::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ce117c9d-4bc8-4529-92b5-32f341614585`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70454::DEBUG::2014-09-25 17:29:42,368::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70454::DEBUG::2014-09-25 17:29:42,369::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70454::DEBUG::2014-09-25 17:29:42,369::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ce117c9d-4bc8-4529-92b5-32f341614585`::Granted request >Thread-70454::DEBUG::2014-09-25 17:29:42,370::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c2cd6203-91fd-453d-add6-cc439802811d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70454::DEBUG::2014-09-25 17:29:42,370::task::993::Storage.TaskManager.Task::(_decref) Task=`c2cd6203-91fd-453d-add6-cc439802811d`::ref 1 aborting False >Thread-70454::INFO::2014-09-25 17:29:42,374::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70454::DEBUG::2014-09-25 17:29:42,375::task::1191::Storage.TaskManager.Task::(prepare) Task=`c2cd6203-91fd-453d-add6-cc439802811d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70454::DEBUG::2014-09-25 17:29:42,375::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2cd6203-91fd-453d-add6-cc439802811d`::moving from state preparing -> state finished >Thread-70454::DEBUG::2014-09-25 17:29:42,375::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70454::DEBUG::2014-09-25 17:29:42,375::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70454::DEBUG::2014-09-25 17:29:42,376::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70454::DEBUG::2014-09-25 17:29:42,376::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70454::DEBUG::2014-09-25 17:29:42,376::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70454::DEBUG::2014-09-25 17:29:42,376::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70454::DEBUG::2014-09-25 17:29:42,377::task::993::Storage.TaskManager.Task::(_decref) Task=`c2cd6203-91fd-453d-add6-cc439802811d`::ref 0 aborting False >Thread-70454::DEBUG::2014-09-25 17:29:42,377::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70454::DEBUG::2014-09-25 17:29:42,378::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:43,003::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:43,004::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70455::DEBUG::2014-09-25 17:29:43,006::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a7e9a05-2e1c-4b90-92af-7ee564eab2f9`::moving from state init -> state preparing >Thread-70455::INFO::2014-09-25 17:29:43,006::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70455::INFO::2014-09-25 17:29:43,007::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0028078', 'lastCheck': '7.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000426093', 'lastCheck': '8.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00250088', 'lastCheck': '2.6', 'valid': True}} >Thread-70455::DEBUG::2014-09-25 17:29:43,007::task::1191::Storage.TaskManager.Task::(prepare) Task=`3a7e9a05-2e1c-4b90-92af-7ee564eab2f9`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0028078', 'lastCheck': '7.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000426093', 'lastCheck': '8.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00250088', 'lastCheck': '2.6', 'valid': True}} >Thread-70455::DEBUG::2014-09-25 17:29:43,007::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a7e9a05-2e1c-4b90-92af-7ee564eab2f9`::moving from state preparing -> state finished >Thread-70455::DEBUG::2014-09-25 17:29:43,008::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70455::DEBUG::2014-09-25 17:29:43,008::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70455::DEBUG::2014-09-25 17:29:43,008::task::993::Storage.TaskManager.Task::(_decref) Task=`3a7e9a05-2e1c-4b90-92af-7ee564eab2f9`::ref 0 aborting False >Thread-70455::DEBUG::2014-09-25 17:29:43,010::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:43,025::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:43,027::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70456::DEBUG::2014-09-25 17:29:43,029::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:44,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:44,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134056 s, 76.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:44,325::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:44,325::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:44,325::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:44,326::__init__::232::IOProcess::(_processLogs) (108389) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:44,327::__init__::232::IOProcess::(_processLogs) (108389) Queuing response >Thread-80::DEBUG::2014-09-25 17:29:44,328::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:29:44,337::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000513767 s, 736 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:44,338::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:44,339::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:44,339::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:44,339::__init__::232::IOProcess::(_processLogs) (108390) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:44,340::__init__::232::IOProcess::(_processLogs) (108390) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:45,251::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:45,252::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:45,252::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:45,252::__init__::232::IOProcess::(_processLogs) (108391) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:45,257::__init__::232::IOProcess::(_processLogs) (108391) Queuing response >Thread-79::DEBUG::2014-09-25 17:29:45,258::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:29:45,276::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00301466 s, 186 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:45,277::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:45,277::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:45,278::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:45,278::__init__::232::IOProcess::(_processLogs) (108392) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:45,280::__init__::232::IOProcess::(_processLogs) (108392) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:45,281::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:45,281::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:45,282::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:45,282::__init__::232::IOProcess::(_processLogs) (108393) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:45,283::__init__::232::IOProcess::(_processLogs) (108393) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:45,284::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:45,284::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:45,284::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:45,284::__init__::232::IOProcess::(_processLogs) (108394) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:45,287::__init__::232::IOProcess::(_processLogs) (108394) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:45,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:45,288::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:45,288::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:45,289::__init__::232::IOProcess::(_processLogs) (108395) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:45,291::__init__::232::IOProcess::(_processLogs) (108395) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:46,076::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:46,079::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70457::DEBUG::2014-09-25 17:29:46,085::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:46,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:46,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0479752 s, 21.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:29:48,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:48,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179937 s, 56.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:49,100::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:49,102::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70458::DEBUG::2014-09-25 17:29:49,104::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:29:50,398::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:50,398::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:50,399::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:50,399::__init__::232::IOProcess::(_processLogs) (108396) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:50,404::__init__::232::IOProcess::(_processLogs) (108396) Queuing response >Thread-81::DEBUG::2014-09-25 17:29:50,406::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:29:50,428::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00611208 s, 60.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:50,429::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:50,429::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:50,429::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:50,430::__init__::232::IOProcess::(_processLogs) (108397) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:50,432::__init__::232::IOProcess::(_processLogs) (108397) Queuing response >Dummy-148::DEBUG::2014-09-25 17:29:50,587::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:50,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142944 s, 71.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:52,117::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:52,118::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70459::DEBUG::2014-09-25 17:29:52,120::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:52,403::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70460::DEBUG::2014-09-25 17:29:52,404::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:29:52,404::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70460::DEBUG::2014-09-25 17:29:52,406::task::595::Storage.TaskManager.Task::(_updateState) Task=`8bdecd28-144b-4d1d-b57f-08b7cd85b378`::moving from state init -> state preparing >Thread-70460::INFO::2014-09-25 17:29:52,407::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70460::INFO::2014-09-25 17:29:52,407::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70460::DEBUG::2014-09-25 17:29:52,408::task::1191::Storage.TaskManager.Task::(prepare) Task=`8bdecd28-144b-4d1d-b57f-08b7cd85b378`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70460::DEBUG::2014-09-25 17:29:52,408::task::595::Storage.TaskManager.Task::(_updateState) Task=`8bdecd28-144b-4d1d-b57f-08b7cd85b378`::moving from state preparing -> state finished >Thread-70460::DEBUG::2014-09-25 17:29:52,408::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70460::DEBUG::2014-09-25 17:29:52,408::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70460::DEBUG::2014-09-25 17:29:52,408::task::993::Storage.TaskManager.Task::(_decref) Task=`8bdecd28-144b-4d1d-b57f-08b7cd85b378`::ref 0 aborting False >Thread-70460::DEBUG::2014-09-25 17:29:52,409::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70460::DEBUG::2014-09-25 17:29:52,409::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:52,420::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:52,421::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70461::DEBUG::2014-09-25 17:29:52,421::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70461::DEBUG::2014-09-25 17:29:52,423::task::595::Storage.TaskManager.Task::(_updateState) Task=`68d8b59e-279c-4c28-9a52-7e0ee9a8055b`::moving from state init -> state preparing >Thread-70461::INFO::2014-09-25 17:29:52,423::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70461::DEBUG::2014-09-25 17:29:52,424::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`08692d16-fbd7-4233-a389-1a147fded1d9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70461::DEBUG::2014-09-25 17:29:52,424::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70461::DEBUG::2014-09-25 17:29:52,425::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70461::DEBUG::2014-09-25 17:29:52,425::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`08692d16-fbd7-4233-a389-1a147fded1d9`::Granted request >Thread-70461::DEBUG::2014-09-25 17:29:52,425::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`68d8b59e-279c-4c28-9a52-7e0ee9a8055b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70461::DEBUG::2014-09-25 17:29:52,426::task::993::Storage.TaskManager.Task::(_decref) Task=`68d8b59e-279c-4c28-9a52-7e0ee9a8055b`::ref 1 aborting False >Thread-70461::INFO::2014-09-25 17:29:52,430::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70461::DEBUG::2014-09-25 17:29:52,430::task::1191::Storage.TaskManager.Task::(prepare) Task=`68d8b59e-279c-4c28-9a52-7e0ee9a8055b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70461::DEBUG::2014-09-25 17:29:52,431::task::595::Storage.TaskManager.Task::(_updateState) Task=`68d8b59e-279c-4c28-9a52-7e0ee9a8055b`::moving from state preparing -> state finished >Thread-70461::DEBUG::2014-09-25 17:29:52,431::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70461::DEBUG::2014-09-25 17:29:52,431::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70461::DEBUG::2014-09-25 17:29:52,432::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70461::DEBUG::2014-09-25 17:29:52,432::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70461::DEBUG::2014-09-25 17:29:52,432::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70461::DEBUG::2014-09-25 17:29:52,432::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70461::DEBUG::2014-09-25 17:29:52,433::task::993::Storage.TaskManager.Task::(_decref) Task=`68d8b59e-279c-4c28-9a52-7e0ee9a8055b`::ref 0 aborting False >Thread-70461::DEBUG::2014-09-25 17:29:52,433::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75198627840', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70461::DEBUG::2014-09-25 17:29:52,434::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:52,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:52,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0208186 s, 49.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:54,342::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:54,342::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:54,343::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:54,343::__init__::232::IOProcess::(_processLogs) (108398) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:54,343::__init__::232::IOProcess::(_processLogs) (108398) Queuing response >Thread-80::DEBUG::2014-09-25 17:29:54,345::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:29:54,354::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000536588 s, 704 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:54,355::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:54,355::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:54,355::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:54,356::__init__::232::IOProcess::(_processLogs) (108399) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:54,356::__init__::232::IOProcess::(_processLogs) (108399) Queuing response >Dummy-148::DEBUG::2014-09-25 17:29:54,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:54,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159817 s, 64.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:55,137::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:55,139::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70462::DEBUG::2014-09-25 17:29:55,141::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:29:55,292::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:55,293::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:55,293::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:55,293::__init__::232::IOProcess::(_processLogs) (108400) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:55,296::__init__::232::IOProcess::(_processLogs) (108400) Queuing response >Thread-79::DEBUG::2014-09-25 17:29:55,298::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:29:55,313::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00384127 s, 146 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:29:55,314::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:55,314::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:55,314::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:55,315::__init__::232::IOProcess::(_processLogs) (108401) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:29:55,318::__init__::232::IOProcess::(_processLogs) (108401) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:55,319::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:55,320::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:55,320::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:55,320::__init__::232::IOProcess::(_processLogs) (108402) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:55,320::__init__::232::IOProcess::(_processLogs) (108402) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:55,321::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:55,322::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:55,323::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:55,323::__init__::232::IOProcess::(_processLogs) (108403) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:55,323::__init__::232::IOProcess::(_processLogs) (108403) Queuing response >Thread-29::DEBUG::2014-09-25 17:29:55,324::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:29:55,324::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:29:55,325::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:29:55,325::__init__::232::IOProcess::(_processLogs) (108404) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:29:55,326::__init__::232::IOProcess::(_processLogs) (108404) Queuing response >Dummy-148::DEBUG::2014-09-25 17:29:56,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:56,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191057 s, 53.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:58,160::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:58,161::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70463::DEBUG::2014-09-25 17:29:58,164::task::595::Storage.TaskManager.Task::(_updateState) Task=`86a030d9-1881-4fed-9eec-d7dc98c536be`::moving from state init -> state preparing >Thread-70463::INFO::2014-09-25 17:29:58,164::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70463::INFO::2014-09-25 17:29:58,165::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00384127', 'lastCheck': '2.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000536588', 'lastCheck': '3.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00611208', 'lastCheck': '7.7', 'valid': True}} >Thread-70463::DEBUG::2014-09-25 17:29:58,165::task::1191::Storage.TaskManager.Task::(prepare) Task=`86a030d9-1881-4fed-9eec-d7dc98c536be`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00384127', 'lastCheck': '2.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000536588', 'lastCheck': '3.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00611208', 'lastCheck': '7.7', 'valid': True}} >Thread-70463::DEBUG::2014-09-25 17:29:58,165::task::595::Storage.TaskManager.Task::(_updateState) Task=`86a030d9-1881-4fed-9eec-d7dc98c536be`::moving from state preparing -> state finished >Thread-70463::DEBUG::2014-09-25 17:29:58,166::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70463::DEBUG::2014-09-25 17:29:58,166::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70463::DEBUG::2014-09-25 17:29:58,166::task::993::Storage.TaskManager.Task::(_decref) Task=`86a030d9-1881-4fed-9eec-d7dc98c536be`::ref 0 aborting False >Thread-70463::DEBUG::2014-09-25 17:29:58,168::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:29:58,186::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:29:58,188::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70464::DEBUG::2014-09-25 17:29:58,189::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:29:58,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:29:58,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0305231 s, 33.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:00,434::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:00,435::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:00,435::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:00,435::__init__::232::IOProcess::(_processLogs) (108405) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:00,444::__init__::232::IOProcess::(_processLogs) (108405) Queuing response >Thread-81::DEBUG::2014-09-25 17:30:00,446::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:30:00,465::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00227874 s, 163 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:00,466::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:00,467::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:00,467::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:00,467::__init__::232::IOProcess::(_processLogs) (108406) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:00,468::__init__::232::IOProcess::(_processLogs) (108406) Queuing response >Dummy-148::DEBUG::2014-09-25 17:30:00,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:00,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134168 s, 76.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:01,251::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:01,252::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70465::DEBUG::2014-09-25 17:30:01,254::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:02,459::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:02,460::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70466::DEBUG::2014-09-25 17:30:02,460::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70466::DEBUG::2014-09-25 17:30:02,463::task::595::Storage.TaskManager.Task::(_updateState) Task=`3de12949-fa2b-4e7e-8771-0b739f70a4c0`::moving from state init -> state preparing >Thread-70466::INFO::2014-09-25 17:30:02,463::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70466::INFO::2014-09-25 17:30:02,463::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70466::DEBUG::2014-09-25 17:30:02,464::task::1191::Storage.TaskManager.Task::(prepare) Task=`3de12949-fa2b-4e7e-8771-0b739f70a4c0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70466::DEBUG::2014-09-25 17:30:02,464::task::595::Storage.TaskManager.Task::(_updateState) Task=`3de12949-fa2b-4e7e-8771-0b739f70a4c0`::moving from state preparing -> state finished >Thread-70466::DEBUG::2014-09-25 17:30:02,464::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70466::DEBUG::2014-09-25 17:30:02,464::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70466::DEBUG::2014-09-25 17:30:02,465::task::993::Storage.TaskManager.Task::(_decref) Task=`3de12949-fa2b-4e7e-8771-0b739f70a4c0`::ref 0 aborting False >Thread-70466::DEBUG::2014-09-25 17:30:02,465::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70466::DEBUG::2014-09-25 17:30:02,465::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:02,476::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:02,477::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70467::DEBUG::2014-09-25 17:30:02,477::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70467::DEBUG::2014-09-25 17:30:02,478::task::595::Storage.TaskManager.Task::(_updateState) Task=`57fd01ad-dffe-443e-8d02-dcdd9efd5c22`::moving from state init -> state preparing >Thread-70467::INFO::2014-09-25 17:30:02,479::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70467::DEBUG::2014-09-25 17:30:02,479::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e67f403c-5615-4d78-8901-e4f328039050`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70467::DEBUG::2014-09-25 17:30:02,480::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70467::DEBUG::2014-09-25 17:30:02,480::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70467::DEBUG::2014-09-25 17:30:02,480::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e67f403c-5615-4d78-8901-e4f328039050`::Granted request >Thread-70467::DEBUG::2014-09-25 17:30:02,481::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`57fd01ad-dffe-443e-8d02-dcdd9efd5c22`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70467::DEBUG::2014-09-25 17:30:02,481::task::993::Storage.TaskManager.Task::(_decref) Task=`57fd01ad-dffe-443e-8d02-dcdd9efd5c22`::ref 1 aborting False >Thread-70467::INFO::2014-09-25 17:30:02,486::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70467::DEBUG::2014-09-25 17:30:02,486::task::1191::Storage.TaskManager.Task::(prepare) Task=`57fd01ad-dffe-443e-8d02-dcdd9efd5c22`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70467::DEBUG::2014-09-25 17:30:02,486::task::595::Storage.TaskManager.Task::(_updateState) Task=`57fd01ad-dffe-443e-8d02-dcdd9efd5c22`::moving from state preparing -> state finished >Thread-70467::DEBUG::2014-09-25 17:30:02,486::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70467::DEBUG::2014-09-25 17:30:02,487::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70467::DEBUG::2014-09-25 17:30:02,487::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70467::DEBUG::2014-09-25 17:30:02,487::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70467::DEBUG::2014-09-25 17:30:02,488::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70467::DEBUG::2014-09-25 17:30:02,488::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70467::DEBUG::2014-09-25 17:30:02,488::task::993::Storage.TaskManager.Task::(_decref) Task=`57fd01ad-dffe-443e-8d02-dcdd9efd5c22`::ref 0 aborting False >Thread-70467::DEBUG::2014-09-25 17:30:02,489::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70467::DEBUG::2014-09-25 17:30:02,489::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:02,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:02,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184115 s, 55.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:04,268::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:04,269::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70468::DEBUG::2014-09-25 17:30:04,271::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:30:04,357::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:04,358::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:04,358::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:04,358::__init__::232::IOProcess::(_processLogs) (108407) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:04,359::__init__::232::IOProcess::(_processLogs) (108407) Queuing response >Thread-80::DEBUG::2014-09-25 17:30:04,360::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:30:04,369::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000440261 s, 859 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:04,370::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:04,370::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:04,371::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:04,371::__init__::232::IOProcess::(_processLogs) (108408) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:04,372::__init__::232::IOProcess::(_processLogs) (108408) Queuing response >Dummy-148::DEBUG::2014-09-25 17:30:04,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:04,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165611 s, 61.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:05,326::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:05,326::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:05,326::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:05,327::__init__::232::IOProcess::(_processLogs) (108409) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:05,327::__init__::232::IOProcess::(_processLogs) (108409) Queuing response >Thread-79::DEBUG::2014-09-25 17:30:05,328::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:30:05,337::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000443359 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:05,338::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:05,339::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:05,339::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:05,339::__init__::232::IOProcess::(_processLogs) (108410) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:05,340::__init__::232::IOProcess::(_processLogs) (108410) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:05,341::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:05,341::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:05,342::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:05,342::__init__::232::IOProcess::(_processLogs) (108411) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:05,343::__init__::232::IOProcess::(_processLogs) (108411) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:05,343::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:05,343::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:05,344::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:05,345::__init__::232::IOProcess::(_processLogs) (108412) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:05,345::__init__::232::IOProcess::(_processLogs) (108412) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:05,345::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:05,346::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:05,346::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:05,346::__init__::232::IOProcess::(_processLogs) (108413) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:05,346::__init__::232::IOProcess::(_processLogs) (108413) Queuing response >Dummy-148::DEBUG::2014-09-25 17:30:06,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:07,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.243706 s, 4.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:07,287::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:07,289::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70469::DEBUG::2014-09-25 17:30:07,291::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:09,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:09,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159179 s, 64.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:10,307::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:10,308::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70470::DEBUG::2014-09-25 17:30:10,310::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:30:10,470::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:10,471::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:10,471::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:10,471::__init__::232::IOProcess::(_processLogs) (108414) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:10,473::__init__::232::IOProcess::(_processLogs) (108414) Queuing response >Thread-81::DEBUG::2014-09-25 17:30:10,474::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:30:10,488::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00359338 s, 103 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:10,489::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:10,489::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:10,490::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:10,490::__init__::232::IOProcess::(_processLogs) (108415) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:10,493::__init__::232::IOProcess::(_processLogs) (108415) Queuing response >Dummy-148::DEBUG::2014-09-25 17:30:11,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:11,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133812 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:12,513::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70471::DEBUG::2014-09-25 17:30:12,514::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:30:12,514::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70471::DEBUG::2014-09-25 17:30:12,517::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f6e379b-47ee-46ff-b728-c48b262f6b6e`::moving from state init -> state preparing >Thread-70471::INFO::2014-09-25 17:30:12,517::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70471::INFO::2014-09-25 17:30:12,518::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70471::DEBUG::2014-09-25 17:30:12,518::task::1191::Storage.TaskManager.Task::(prepare) Task=`4f6e379b-47ee-46ff-b728-c48b262f6b6e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70471::DEBUG::2014-09-25 17:30:12,518::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f6e379b-47ee-46ff-b728-c48b262f6b6e`::moving from state preparing -> state finished >Thread-70471::DEBUG::2014-09-25 17:30:12,518::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70471::DEBUG::2014-09-25 17:30:12,518::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70471::DEBUG::2014-09-25 17:30:12,519::task::993::Storage.TaskManager.Task::(_decref) Task=`4f6e379b-47ee-46ff-b728-c48b262f6b6e`::ref 0 aborting False >Thread-70471::DEBUG::2014-09-25 17:30:12,519::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70471::DEBUG::2014-09-25 17:30:12,519::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:12,530::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70472::DEBUG::2014-09-25 17:30:12,531::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:30:12,531::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70472::DEBUG::2014-09-25 17:30:12,533::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c3e98d5-46b7-4cf6-859e-be815bad989d`::moving from state init -> state preparing >Thread-70472::INFO::2014-09-25 17:30:12,534::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70472::DEBUG::2014-09-25 17:30:12,534::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ec58245a-be25-4b6c-84e8-dec02a6c3a46`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70472::DEBUG::2014-09-25 17:30:12,535::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70472::DEBUG::2014-09-25 17:30:12,535::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70472::DEBUG::2014-09-25 17:30:12,535::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ec58245a-be25-4b6c-84e8-dec02a6c3a46`::Granted request >Thread-70472::DEBUG::2014-09-25 17:30:12,536::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3c3e98d5-46b7-4cf6-859e-be815bad989d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70472::DEBUG::2014-09-25 17:30:12,536::task::993::Storage.TaskManager.Task::(_decref) Task=`3c3e98d5-46b7-4cf6-859e-be815bad989d`::ref 1 aborting False >Thread-70472::INFO::2014-09-25 17:30:12,541::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70472::DEBUG::2014-09-25 17:30:12,541::task::1191::Storage.TaskManager.Task::(prepare) Task=`3c3e98d5-46b7-4cf6-859e-be815bad989d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70472::DEBUG::2014-09-25 17:30:12,541::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c3e98d5-46b7-4cf6-859e-be815bad989d`::moving from state preparing -> state finished >Thread-70472::DEBUG::2014-09-25 17:30:12,541::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70472::DEBUG::2014-09-25 17:30:12,542::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70472::DEBUG::2014-09-25 17:30:12,542::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70472::DEBUG::2014-09-25 17:30:12,542::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70472::DEBUG::2014-09-25 17:30:12,543::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70472::DEBUG::2014-09-25 17:30:12,543::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70472::DEBUG::2014-09-25 17:30:12,543::task::993::Storage.TaskManager.Task::(_decref) Task=`3c3e98d5-46b7-4cf6-859e-be815bad989d`::ref 0 aborting False >Thread-70472::DEBUG::2014-09-25 17:30:12,544::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70472::DEBUG::2014-09-25 17:30:12,544::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:13,323::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:13,324::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70473::DEBUG::2014-09-25 17:30:13,326::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2b3612e-96fc-4035-80b6-d1cc73766706`::moving from state init -> state preparing >Thread-70473::INFO::2014-09-25 17:30:13,326::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70473::INFO::2014-09-25 17:30:13,327::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443359', 'lastCheck': '8.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000440261', 'lastCheck': '9.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00359338', 'lastCheck': '2.8', 'valid': True}} >Thread-70473::DEBUG::2014-09-25 17:30:13,327::task::1191::Storage.TaskManager.Task::(prepare) Task=`c2b3612e-96fc-4035-80b6-d1cc73766706`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443359', 'lastCheck': '8.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000440261', 'lastCheck': '9.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00359338', 'lastCheck': '2.8', 'valid': True}} >Thread-70473::DEBUG::2014-09-25 17:30:13,327::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2b3612e-96fc-4035-80b6-d1cc73766706`::moving from state preparing -> state finished >Thread-70473::DEBUG::2014-09-25 17:30:13,328::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70473::DEBUG::2014-09-25 17:30:13,328::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70473::DEBUG::2014-09-25 17:30:13,328::task::993::Storage.TaskManager.Task::(_decref) Task=`c2b3612e-96fc-4035-80b6-d1cc73766706`::ref 0 aborting False >Thread-70473::DEBUG::2014-09-25 17:30:13,330::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:13,345::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:13,346::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70474::DEBUG::2014-09-25 17:30:13,348::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:13,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:13,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021958 s, 46.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:14,374::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:14,374::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:14,374::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:14,375::__init__::232::IOProcess::(_processLogs) (108416) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:14,375::__init__::232::IOProcess::(_processLogs) (108416) Queuing response >Thread-80::DEBUG::2014-09-25 17:30:14,376::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:30:14,385::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000442382 s, 854 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:14,386::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:14,386::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:14,387::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:14,388::__init__::232::IOProcess::(_processLogs) (108417) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:14,389::__init__::232::IOProcess::(_processLogs) (108417) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:15,348::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:15,348::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:15,348::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:15,349::__init__::232::IOProcess::(_processLogs) (108418) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:15,364::__init__::232::IOProcess::(_processLogs) (108418) Queuing response >Thread-79::DEBUG::2014-09-25 17:30:15,365::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:30:15,383::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000895568 s, 628 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:15,384::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:15,384::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:15,385::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:15,385::__init__::232::IOProcess::(_processLogs) (108419) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:15,390::__init__::232::IOProcess::(_processLogs) (108419) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:15,391::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:15,392::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:15,392::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:15,392::__init__::232::IOProcess::(_processLogs) (108420) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:15,392::__init__::232::IOProcess::(_processLogs) (108420) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:15,393::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:15,394::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:15,394::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:15,394::__init__::232::IOProcess::(_processLogs) (108421) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:15,395::__init__::232::IOProcess::(_processLogs) (108421) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:15,396::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:15,396::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:15,396::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:15,396::__init__::232::IOProcess::(_processLogs) (108422) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:15,397::__init__::232::IOProcess::(_processLogs) (108422) Queuing response >Dummy-148::DEBUG::2014-09-25 17:30:15,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:15,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0196338 s, 52.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:16,395::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:16,400::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70475::DEBUG::2014-09-25 17:30:16,409::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:17,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:17,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164497 s, 62.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:19,425::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:19,427::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70476::DEBUG::2014-09-25 17:30:19,429::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:19,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:19,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131341 s, 78.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:20,495::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:20,496::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:20,496::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:20,496::__init__::232::IOProcess::(_processLogs) (108423) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:20,497::__init__::232::IOProcess::(_processLogs) (108423) Queuing response >Thread-81::DEBUG::2014-09-25 17:30:20,498::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:30:20,507::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000357602 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:20,508::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:20,508::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:20,508::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:20,509::__init__::232::IOProcess::(_processLogs) (108424) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:20,509::__init__::232::IOProcess::(_processLogs) (108424) Queuing response >Dummy-148::DEBUG::2014-09-25 17:30:21,555::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:21,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158954 s, 64.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:22,444::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:22,445::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70477::DEBUG::2014-09-25 17:30:22,447::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:22,568::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70478::DEBUG::2014-09-25 17:30:22,569::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:30:22,569::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70478::DEBUG::2014-09-25 17:30:22,571::task::595::Storage.TaskManager.Task::(_updateState) Task=`20777c39-12ee-436a-87f1-bfd50430ce94`::moving from state init -> state preparing >Thread-70478::INFO::2014-09-25 17:30:22,572::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70478::INFO::2014-09-25 17:30:22,572::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70478::DEBUG::2014-09-25 17:30:22,572::task::1191::Storage.TaskManager.Task::(prepare) Task=`20777c39-12ee-436a-87f1-bfd50430ce94`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70478::DEBUG::2014-09-25 17:30:22,573::task::595::Storage.TaskManager.Task::(_updateState) Task=`20777c39-12ee-436a-87f1-bfd50430ce94`::moving from state preparing -> state finished >Thread-70478::DEBUG::2014-09-25 17:30:22,573::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70478::DEBUG::2014-09-25 17:30:22,573::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70478::DEBUG::2014-09-25 17:30:22,573::task::993::Storage.TaskManager.Task::(_decref) Task=`20777c39-12ee-436a-87f1-bfd50430ce94`::ref 0 aborting False >Thread-70478::DEBUG::2014-09-25 17:30:22,574::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70478::DEBUG::2014-09-25 17:30:22,574::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:22,591::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70479::DEBUG::2014-09-25 17:30:22,592::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:30:22,592::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70479::DEBUG::2014-09-25 17:30:22,594::task::595::Storage.TaskManager.Task::(_updateState) Task=`0754d22f-be65-494c-ac10-90425085b18c`::moving from state init -> state preparing >Thread-70479::INFO::2014-09-25 17:30:22,595::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70479::DEBUG::2014-09-25 17:30:22,595::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2ee2e460-9112-40a0-a7e6-c3ddd435806c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70479::DEBUG::2014-09-25 17:30:22,596::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70479::DEBUG::2014-09-25 17:30:22,596::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70479::DEBUG::2014-09-25 17:30:22,596::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2ee2e460-9112-40a0-a7e6-c3ddd435806c`::Granted request >Thread-70479::DEBUG::2014-09-25 17:30:22,597::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0754d22f-be65-494c-ac10-90425085b18c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70479::DEBUG::2014-09-25 17:30:22,597::task::993::Storage.TaskManager.Task::(_decref) Task=`0754d22f-be65-494c-ac10-90425085b18c`::ref 1 aborting False >Thread-70479::INFO::2014-09-25 17:30:22,601::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70479::DEBUG::2014-09-25 17:30:22,602::task::1191::Storage.TaskManager.Task::(prepare) Task=`0754d22f-be65-494c-ac10-90425085b18c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70479::DEBUG::2014-09-25 17:30:22,602::task::595::Storage.TaskManager.Task::(_updateState) Task=`0754d22f-be65-494c-ac10-90425085b18c`::moving from state preparing -> state finished >Thread-70479::DEBUG::2014-09-25 17:30:22,602::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70479::DEBUG::2014-09-25 17:30:22,602::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70479::DEBUG::2014-09-25 17:30:22,603::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70479::DEBUG::2014-09-25 17:30:22,603::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70479::DEBUG::2014-09-25 17:30:22,603::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70479::DEBUG::2014-09-25 17:30:22,604::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70479::DEBUG::2014-09-25 17:30:22,604::task::993::Storage.TaskManager.Task::(_decref) Task=`0754d22f-be65-494c-ac10-90425085b18c`::ref 0 aborting False >Thread-70479::DEBUG::2014-09-25 17:30:22,604::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70479::DEBUG::2014-09-25 17:30:22,605::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:23,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:23,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147653 s, 69.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:24,389::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:24,390::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:24,390::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:24,390::__init__::232::IOProcess::(_processLogs) (108425) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:24,391::__init__::232::IOProcess::(_processLogs) (108425) Queuing response >Thread-80::DEBUG::2014-09-25 17:30:24,392::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:30:24,401::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000448204 s, 843 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:24,402::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:24,402::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:24,403::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:24,404::__init__::232::IOProcess::(_processLogs) (108426) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:24,404::__init__::232::IOProcess::(_processLogs) (108426) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:25,399::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:25,399::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:25,399::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:25,400::__init__::232::IOProcess::(_processLogs) (108427) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:25,404::__init__::232::IOProcess::(_processLogs) (108427) Queuing response >Thread-79::DEBUG::2014-09-25 17:30:25,405::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:30:25,417::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00215257 s, 261 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:25,418::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:25,418::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:25,419::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:25,419::__init__::232::IOProcess::(_processLogs) (108428) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:25,420::__init__::232::IOProcess::(_processLogs) (108428) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:25,421::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:25,421::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:25,421::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:25,422::__init__::232::IOProcess::(_processLogs) (108429) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:25,422::__init__::232::IOProcess::(_processLogs) (108429) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:25,423::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:25,423::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:25,424::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:25,424::__init__::232::IOProcess::(_processLogs) (108430) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:25,424::__init__::232::IOProcess::(_processLogs) (108430) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:25,425::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:25,425::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:25,426::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:25,426::__init__::232::IOProcess::(_processLogs) (108431) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:25,426::__init__::232::IOProcess::(_processLogs) (108431) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:25,463::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:25,465::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70480::DEBUG::2014-09-25 17:30:25,467::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:25,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:25,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169387 s, 60.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:30:27,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:27,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0176172 s, 58.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:28,479::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:28,481::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70481::DEBUG::2014-09-25 17:30:28,483::task::595::Storage.TaskManager.Task::(_updateState) Task=`ddde9793-08d3-4327-b931-ebe8480bb615`::moving from state init -> state preparing >Thread-70481::INFO::2014-09-25 17:30:28,483::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70481::INFO::2014-09-25 17:30:28,484::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00215257', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000448204', 'lastCheck': '4.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000357602', 'lastCheck': '8.0', 'valid': True}} >Thread-70481::DEBUG::2014-09-25 17:30:28,484::task::1191::Storage.TaskManager.Task::(prepare) Task=`ddde9793-08d3-4327-b931-ebe8480bb615`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00215257', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000448204', 'lastCheck': '4.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000357602', 'lastCheck': '8.0', 'valid': True}} >Thread-70481::DEBUG::2014-09-25 17:30:28,484::task::595::Storage.TaskManager.Task::(_updateState) Task=`ddde9793-08d3-4327-b931-ebe8480bb615`::moving from state preparing -> state finished >Thread-70481::DEBUG::2014-09-25 17:30:28,485::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70481::DEBUG::2014-09-25 17:30:28,485::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70481::DEBUG::2014-09-25 17:30:28,485::task::993::Storage.TaskManager.Task::(_decref) Task=`ddde9793-08d3-4327-b931-ebe8480bb615`::ref 0 aborting False >Thread-70481::DEBUG::2014-09-25 17:30:28,488::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:28,504::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:28,505::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70482::DEBUG::2014-09-25 17:30:28,507::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:29,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:29,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169254 s, 60.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:30,512::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:30,512::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:30,512::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:30,512::__init__::232::IOProcess::(_processLogs) (108432) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:30,513::__init__::232::IOProcess::(_processLogs) (108432) Queuing response >Thread-81::DEBUG::2014-09-25 17:30:30,514::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:30:30,525::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000715012 s, 519 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:30,526::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:30,526::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:30,527::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:30,527::__init__::232::IOProcess::(_processLogs) (108433) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:30,528::__init__::232::IOProcess::(_processLogs) (108433) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:31,552::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:31,553::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70483::DEBUG::2014-09-25 17:30:31,555::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:31,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:31,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157707 s, 64.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:32,633::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70484::DEBUG::2014-09-25 17:30:32,634::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:30:32,634::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70484::DEBUG::2014-09-25 17:30:32,637::task::595::Storage.TaskManager.Task::(_updateState) Task=`7a8b1a5e-b710-4d32-bdb5-a059c66d75d3`::moving from state init -> state preparing >Thread-70484::INFO::2014-09-25 17:30:32,637::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70484::INFO::2014-09-25 17:30:32,637::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70484::DEBUG::2014-09-25 17:30:32,638::task::1191::Storage.TaskManager.Task::(prepare) Task=`7a8b1a5e-b710-4d32-bdb5-a059c66d75d3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70484::DEBUG::2014-09-25 17:30:32,638::task::595::Storage.TaskManager.Task::(_updateState) Task=`7a8b1a5e-b710-4d32-bdb5-a059c66d75d3`::moving from state preparing -> state finished >Thread-70484::DEBUG::2014-09-25 17:30:32,638::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70484::DEBUG::2014-09-25 17:30:32,638::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70484::DEBUG::2014-09-25 17:30:32,639::task::993::Storage.TaskManager.Task::(_decref) Task=`7a8b1a5e-b710-4d32-bdb5-a059c66d75d3`::ref 0 aborting False >Thread-70484::DEBUG::2014-09-25 17:30:32,639::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70484::DEBUG::2014-09-25 17:30:32,639::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:32,654::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:32,655::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70485::DEBUG::2014-09-25 17:30:32,655::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70485::DEBUG::2014-09-25 17:30:32,657::task::595::Storage.TaskManager.Task::(_updateState) Task=`e5f64b9d-c05e-4944-92bd-f6f131c5046c`::moving from state init -> state preparing >Thread-70485::INFO::2014-09-25 17:30:32,657::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70485::DEBUG::2014-09-25 17:30:32,658::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`541f19e1-fff5-4e35-a1cb-33ff26092604`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70485::DEBUG::2014-09-25 17:30:32,658::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70485::DEBUG::2014-09-25 17:30:32,658::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70485::DEBUG::2014-09-25 17:30:32,659::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`541f19e1-fff5-4e35-a1cb-33ff26092604`::Granted request >Thread-70485::DEBUG::2014-09-25 17:30:32,659::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e5f64b9d-c05e-4944-92bd-f6f131c5046c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70485::DEBUG::2014-09-25 17:30:32,660::task::993::Storage.TaskManager.Task::(_decref) Task=`e5f64b9d-c05e-4944-92bd-f6f131c5046c`::ref 1 aborting False >Thread-70485::INFO::2014-09-25 17:30:32,664::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70485::DEBUG::2014-09-25 17:30:32,664::task::1191::Storage.TaskManager.Task::(prepare) Task=`e5f64b9d-c05e-4944-92bd-f6f131c5046c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70485::DEBUG::2014-09-25 17:30:32,664::task::595::Storage.TaskManager.Task::(_updateState) Task=`e5f64b9d-c05e-4944-92bd-f6f131c5046c`::moving from state preparing -> state finished >Thread-70485::DEBUG::2014-09-25 17:30:32,665::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70485::DEBUG::2014-09-25 17:30:32,665::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70485::DEBUG::2014-09-25 17:30:32,665::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70485::DEBUG::2014-09-25 17:30:32,666::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70485::DEBUG::2014-09-25 17:30:32,666::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70485::DEBUG::2014-09-25 17:30:32,666::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70485::DEBUG::2014-09-25 17:30:32,666::task::993::Storage.TaskManager.Task::(_decref) Task=`e5f64b9d-c05e-4944-92bd-f6f131c5046c`::ref 0 aborting False >Thread-70485::DEBUG::2014-09-25 17:30:32,667::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70485::DEBUG::2014-09-25 17:30:32,667::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:33,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:33,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168832 s, 60.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:34,405::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:34,405::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:34,406::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:34,406::__init__::232::IOProcess::(_processLogs) (108434) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:34,407::__init__::232::IOProcess::(_processLogs) (108434) Queuing response >Thread-80::DEBUG::2014-09-25 17:30:34,408::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:30:34,417::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000776457 s, 487 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:34,418::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:34,418::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:34,419::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:34,419::__init__::232::IOProcess::(_processLogs) (108435) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:34,420::__init__::232::IOProcess::(_processLogs) (108435) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:34,569::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:34,570::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70486::DEBUG::2014-09-25 17:30:34,572::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:30:35,427::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:35,428::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:35,428::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:35,428::__init__::232::IOProcess::(_processLogs) (108436) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:35,431::__init__::232::IOProcess::(_processLogs) (108436) Queuing response >Thread-79::DEBUG::2014-09-25 17:30:35,432::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:30:35,452::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00769368 s, 73.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:35,453::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:35,453::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:35,453::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:35,454::__init__::232::IOProcess::(_processLogs) (108437) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:35,458::__init__::232::IOProcess::(_processLogs) (108437) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:35,459::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:35,459::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:35,460::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:35,460::__init__::232::IOProcess::(_processLogs) (108438) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:35,460::__init__::232::IOProcess::(_processLogs) (108438) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:35,461::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:35,461::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:35,462::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:35,462::__init__::232::IOProcess::(_processLogs) (108439) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:35,463::__init__::232::IOProcess::(_processLogs) (108439) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:35,463::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:35,464::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:35,464::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:35,465::__init__::232::IOProcess::(_processLogs) (108440) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:35,465::__init__::232::IOProcess::(_processLogs) (108440) Queuing response >Dummy-148::DEBUG::2014-09-25 17:30:35,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:35,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132482 s, 77.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:37,584::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:37,586::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70487::DEBUG::2014-09-25 17:30:37,588::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:37,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:37,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169386 s, 60.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:30:39,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:40,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160895 s, 63.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:40,530::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:40,530::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:40,531::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:40,531::__init__::232::IOProcess::(_processLogs) (108441) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:40,533::__init__::232::IOProcess::(_processLogs) (108441) Queuing response >Thread-81::DEBUG::2014-09-25 17:30:40,534::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:30:40,556::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00897308 s, 41.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:40,557::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:40,558::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:40,558::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:40,558::__init__::232::IOProcess::(_processLogs) (108442) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:40,560::__init__::232::IOProcess::(_processLogs) (108442) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:40,603::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:40,605::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70488::DEBUG::2014-09-25 17:30:40,607::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:42,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:42,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161736 s, 63.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:42,702::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70489::DEBUG::2014-09-25 17:30:42,703::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:30:42,703::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70489::DEBUG::2014-09-25 17:30:42,706::task::595::Storage.TaskManager.Task::(_updateState) Task=`836f67ec-103f-45b6-9ce2-491fe4062960`::moving from state init -> state preparing >Thread-70489::INFO::2014-09-25 17:30:42,706::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70489::INFO::2014-09-25 17:30:42,707::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70489::DEBUG::2014-09-25 17:30:42,707::task::1191::Storage.TaskManager.Task::(prepare) Task=`836f67ec-103f-45b6-9ce2-491fe4062960`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70489::DEBUG::2014-09-25 17:30:42,707::task::595::Storage.TaskManager.Task::(_updateState) Task=`836f67ec-103f-45b6-9ce2-491fe4062960`::moving from state preparing -> state finished >Thread-70489::DEBUG::2014-09-25 17:30:42,707::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70489::DEBUG::2014-09-25 17:30:42,708::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70489::DEBUG::2014-09-25 17:30:42,708::task::993::Storage.TaskManager.Task::(_decref) Task=`836f67ec-103f-45b6-9ce2-491fe4062960`::ref 0 aborting False >Thread-70489::DEBUG::2014-09-25 17:30:42,708::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70489::DEBUG::2014-09-25 17:30:42,709::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:42,726::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70490::DEBUG::2014-09-25 17:30:42,728::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:30:42,728::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70490::DEBUG::2014-09-25 17:30:42,730::task::595::Storage.TaskManager.Task::(_updateState) Task=`f2012f43-4a78-4944-bb8e-cf9a41ac1624`::moving from state init -> state preparing >Thread-70490::INFO::2014-09-25 17:30:42,730::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70490::DEBUG::2014-09-25 17:30:42,731::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3fb87388-f7ca-4b81-ad51-0b280ffa28ea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70490::DEBUG::2014-09-25 17:30:42,731::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70490::DEBUG::2014-09-25 17:30:42,731::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70490::DEBUG::2014-09-25 17:30:42,732::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3fb87388-f7ca-4b81-ad51-0b280ffa28ea`::Granted request >Thread-70490::DEBUG::2014-09-25 17:30:42,732::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f2012f43-4a78-4944-bb8e-cf9a41ac1624`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70490::DEBUG::2014-09-25 17:30:42,732::task::993::Storage.TaskManager.Task::(_decref) Task=`f2012f43-4a78-4944-bb8e-cf9a41ac1624`::ref 1 aborting False >Thread-70490::INFO::2014-09-25 17:30:42,737::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70490::DEBUG::2014-09-25 17:30:42,737::task::1191::Storage.TaskManager.Task::(prepare) Task=`f2012f43-4a78-4944-bb8e-cf9a41ac1624`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70490::DEBUG::2014-09-25 17:30:42,737::task::595::Storage.TaskManager.Task::(_updateState) Task=`f2012f43-4a78-4944-bb8e-cf9a41ac1624`::moving from state preparing -> state finished >Thread-70490::DEBUG::2014-09-25 17:30:42,738::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70490::DEBUG::2014-09-25 17:30:42,738::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70490::DEBUG::2014-09-25 17:30:42,738::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70490::DEBUG::2014-09-25 17:30:42,739::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70490::DEBUG::2014-09-25 17:30:42,739::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70490::DEBUG::2014-09-25 17:30:42,739::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70490::DEBUG::2014-09-25 17:30:42,739::task::993::Storage.TaskManager.Task::(_decref) Task=`f2012f43-4a78-4944-bb8e-cf9a41ac1624`::ref 0 aborting False >Thread-70490::DEBUG::2014-09-25 17:30:42,740::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70490::DEBUG::2014-09-25 17:30:42,740::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:43,621::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:43,623::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70491::DEBUG::2014-09-25 17:30:43,625::task::595::Storage.TaskManager.Task::(_updateState) Task=`ebf3ef78-e301-47d4-bdd7-bcf87838df79`::moving from state init -> state preparing >Thread-70491::INFO::2014-09-25 17:30:43,626::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70491::INFO::2014-09-25 17:30:43,626::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00769368', 'lastCheck': '8.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000776457', 'lastCheck': '9.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00897308', 'lastCheck': '3.1', 'valid': True}} >Thread-70491::DEBUG::2014-09-25 17:30:43,626::task::1191::Storage.TaskManager.Task::(prepare) Task=`ebf3ef78-e301-47d4-bdd7-bcf87838df79`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00769368', 'lastCheck': '8.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000776457', 'lastCheck': '9.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00897308', 'lastCheck': '3.1', 'valid': True}} >Thread-70491::DEBUG::2014-09-25 17:30:43,627::task::595::Storage.TaskManager.Task::(_updateState) Task=`ebf3ef78-e301-47d4-bdd7-bcf87838df79`::moving from state preparing -> state finished >Thread-70491::DEBUG::2014-09-25 17:30:43,627::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70491::DEBUG::2014-09-25 17:30:43,627::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70491::DEBUG::2014-09-25 17:30:43,627::task::993::Storage.TaskManager.Task::(_decref) Task=`ebf3ef78-e301-47d4-bdd7-bcf87838df79`::ref 0 aborting False >Thread-70491::DEBUG::2014-09-25 17:30:43,630::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:43,651::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:43,652::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70492::DEBUG::2014-09-25 17:30:43,654::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:44,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:44,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171292 s, 59.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:44,422::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:44,422::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:44,422::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:44,423::__init__::232::IOProcess::(_processLogs) (108443) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:44,423::__init__::232::IOProcess::(_processLogs) (108443) Queuing response >Thread-80::DEBUG::2014-09-25 17:30:44,425::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:30:44,434::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000465071 s, 813 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:44,435::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:44,435::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:44,436::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:44,437::__init__::232::IOProcess::(_processLogs) (108444) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:44,438::__init__::232::IOProcess::(_processLogs) (108444) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:45,465::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:45,465::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:45,466::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:45,466::__init__::232::IOProcess::(_processLogs) (108445) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:45,468::__init__::232::IOProcess::(_processLogs) (108445) Queuing response >Thread-79::DEBUG::2014-09-25 17:30:45,469::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:30:45,487::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00430204 s, 131 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:45,489::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:45,489::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:45,489::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:45,490::__init__::232::IOProcess::(_processLogs) (108446) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:45,490::__init__::232::IOProcess::(_processLogs) (108446) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:45,492::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:45,492::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:45,492::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:45,493::__init__::232::IOProcess::(_processLogs) (108447) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:45,499::__init__::232::IOProcess::(_processLogs) (108447) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:45,500::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:45,500::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:45,501::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:45,501::__init__::232::IOProcess::(_processLogs) (108448) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:45,509::__init__::232::IOProcess::(_processLogs) (108448) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:45,510::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:45,510::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:45,510::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:45,510::__init__::232::IOProcess::(_processLogs) (108449) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:45,517::__init__::232::IOProcess::(_processLogs) (108449) Queuing response >Dummy-148::DEBUG::2014-09-25 17:30:46,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:46,189::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0273106 s, 37.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:46,707::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:46,716::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70493::DEBUG::2014-09-25 17:30:46,724::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:48,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:48,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211438 s, 48.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:49,744::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:49,745::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70494::DEBUG::2014-09-25 17:30:49,748::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:50,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:50,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0222343 s, 46.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:50,562::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:50,563::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:50,563::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:50,563::__init__::232::IOProcess::(_processLogs) (108450) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:50,565::__init__::232::IOProcess::(_processLogs) (108450) Queuing response >Thread-81::DEBUG::2014-09-25 17:30:50,566::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:30:50,585::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00339239 s, 109 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:50,586::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:50,587::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:50,587::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:50,587::__init__::232::IOProcess::(_processLogs) (108451) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:50,591::__init__::232::IOProcess::(_processLogs) (108451) Queuing response >Dummy-148::DEBUG::2014-09-25 17:30:52,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:52,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146159 s, 70.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:52,761::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:52,762::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70495::DEBUG::2014-09-25 17:30:52,764::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:52,775::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70496::DEBUG::2014-09-25 17:30:52,776::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:30:52,776::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70496::DEBUG::2014-09-25 17:30:52,779::task::595::Storage.TaskManager.Task::(_updateState) Task=`df9d53a6-3001-4d36-a4f6-0eb86e3824f9`::moving from state init -> state preparing >Thread-70496::INFO::2014-09-25 17:30:52,780::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70496::INFO::2014-09-25 17:30:52,781::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70496::DEBUG::2014-09-25 17:30:52,781::task::1191::Storage.TaskManager.Task::(prepare) Task=`df9d53a6-3001-4d36-a4f6-0eb86e3824f9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70496::DEBUG::2014-09-25 17:30:52,781::task::595::Storage.TaskManager.Task::(_updateState) Task=`df9d53a6-3001-4d36-a4f6-0eb86e3824f9`::moving from state preparing -> state finished >Thread-70496::DEBUG::2014-09-25 17:30:52,781::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70496::DEBUG::2014-09-25 17:30:52,782::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70496::DEBUG::2014-09-25 17:30:52,782::task::993::Storage.TaskManager.Task::(_decref) Task=`df9d53a6-3001-4d36-a4f6-0eb86e3824f9`::ref 0 aborting False >Thread-70496::DEBUG::2014-09-25 17:30:52,782::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70496::DEBUG::2014-09-25 17:30:52,782::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:52,794::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:52,795::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70497::DEBUG::2014-09-25 17:30:52,795::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70497::DEBUG::2014-09-25 17:30:52,797::task::595::Storage.TaskManager.Task::(_updateState) Task=`e1723b67-a103-4354-bba9-cc6cd1449586`::moving from state init -> state preparing >Thread-70497::INFO::2014-09-25 17:30:52,797::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70497::DEBUG::2014-09-25 17:30:52,798::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9630b494-12f6-4b2e-9a40-ed932d285439`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70497::DEBUG::2014-09-25 17:30:52,798::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70497::DEBUG::2014-09-25 17:30:52,799::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70497::DEBUG::2014-09-25 17:30:52,799::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9630b494-12f6-4b2e-9a40-ed932d285439`::Granted request >Thread-70497::DEBUG::2014-09-25 17:30:52,800::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e1723b67-a103-4354-bba9-cc6cd1449586`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70497::DEBUG::2014-09-25 17:30:52,800::task::993::Storage.TaskManager.Task::(_decref) Task=`e1723b67-a103-4354-bba9-cc6cd1449586`::ref 1 aborting False >Thread-70497::INFO::2014-09-25 17:30:52,804::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70497::DEBUG::2014-09-25 17:30:52,804::task::1191::Storage.TaskManager.Task::(prepare) Task=`e1723b67-a103-4354-bba9-cc6cd1449586`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70497::DEBUG::2014-09-25 17:30:52,805::task::595::Storage.TaskManager.Task::(_updateState) Task=`e1723b67-a103-4354-bba9-cc6cd1449586`::moving from state preparing -> state finished >Thread-70497::DEBUG::2014-09-25 17:30:52,805::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70497::DEBUG::2014-09-25 17:30:52,805::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70497::DEBUG::2014-09-25 17:30:52,805::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70497::DEBUG::2014-09-25 17:30:52,806::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70497::DEBUG::2014-09-25 17:30:52,806::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70497::DEBUG::2014-09-25 17:30:52,806::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70497::DEBUG::2014-09-25 17:30:52,807::task::993::Storage.TaskManager.Task::(_decref) Task=`e1723b67-a103-4354-bba9-cc6cd1449586`::ref 0 aborting False >Thread-70497::DEBUG::2014-09-25 17:30:52,807::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70497::DEBUG::2014-09-25 17:30:52,808::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:30:54,438::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:54,439::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:54,439::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:30:54,441::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:30:54,441::__init__::232::IOProcess::(_processLogs) (108452) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:54,449::__init__::232::IOProcess::(_processLogs) (108452) Queuing response >Thread-80::DEBUG::2014-09-25 17:30:54,450::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00072522 s, 521 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:54,451::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:54,451::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:54,452::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:54,452::__init__::232::IOProcess::(_processLogs) (108453) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:54,454::__init__::232::IOProcess::(_processLogs) (108453) Queuing response >Dummy-148::DEBUG::2014-09-25 17:30:54,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:54,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184123 s, 55.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:55,519::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:55,519::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:55,519::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:55,520::__init__::232::IOProcess::(_processLogs) (108454) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:55,524::__init__::232::IOProcess::(_processLogs) (108454) Queuing response >Thread-79::DEBUG::2014-09-25 17:30:55,525::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:30:55,546::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00859935 s, 65.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:30:55,548::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:55,548::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:55,548::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:55,548::__init__::232::IOProcess::(_processLogs) (108455) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:30:55,555::__init__::232::IOProcess::(_processLogs) (108455) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:55,556::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:55,556::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:55,557::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:55,557::__init__::232::IOProcess::(_processLogs) (108456) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:55,558::__init__::232::IOProcess::(_processLogs) (108456) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:55,558::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:55,558::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:55,559::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:55,560::__init__::232::IOProcess::(_processLogs) (108457) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:55,560::__init__::232::IOProcess::(_processLogs) (108457) Queuing response >Thread-29::DEBUG::2014-09-25 17:30:55,560::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:30:55,561::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:30:55,562::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:30:55,562::__init__::232::IOProcess::(_processLogs) (108458) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:30:55,562::__init__::232::IOProcess::(_processLogs) (108458) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:55,781::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:55,782::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70498::DEBUG::2014-09-25 17:30:55,784::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:30:56,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:56,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132699 s, 77.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:30:58,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:30:58,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136083 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:58,802::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:58,803::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70499::DEBUG::2014-09-25 17:30:58,806::task::595::Storage.TaskManager.Task::(_updateState) Task=`22576ed3-3501-4adf-9226-03d1bcbde353`::moving from state init -> state preparing >Thread-70499::INFO::2014-09-25 17:30:58,806::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70499::INFO::2014-09-25 17:30:58,807::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00859935', 'lastCheck': '3.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00072522', 'lastCheck': '4.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00339239', 'lastCheck': '8.2', 'valid': True}} >Thread-70499::DEBUG::2014-09-25 17:30:58,807::task::1191::Storage.TaskManager.Task::(prepare) Task=`22576ed3-3501-4adf-9226-03d1bcbde353`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00859935', 'lastCheck': '3.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00072522', 'lastCheck': '4.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00339239', 'lastCheck': '8.2', 'valid': True}} >Thread-70499::DEBUG::2014-09-25 17:30:58,807::task::595::Storage.TaskManager.Task::(_updateState) Task=`22576ed3-3501-4adf-9226-03d1bcbde353`::moving from state preparing -> state finished >Thread-70499::DEBUG::2014-09-25 17:30:58,808::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70499::DEBUG::2014-09-25 17:30:58,808::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70499::DEBUG::2014-09-25 17:30:58,808::task::993::Storage.TaskManager.Task::(_decref) Task=`22576ed3-3501-4adf-9226-03d1bcbde353`::ref 0 aborting False >Thread-70499::DEBUG::2014-09-25 17:30:58,810::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:30:58,830::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:30:58,831::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70500::DEBUG::2014-09-25 17:30:58,833::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:31:00,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:00,593::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:00,593::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:00,593::__init__::232::IOProcess::(_processLogs) (108459) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:00,608::__init__::232::IOProcess::(_processLogs) (108459) Queuing response >Thread-81::DEBUG::2014-09-25 17:31:00,609::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:31:00,644::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00801088 s, 46.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:00,646::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:00,646::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:00,646::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:00,647::__init__::232::IOProcess::(_processLogs) (108460) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:00,649::__init__::232::IOProcess::(_processLogs) (108460) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:00,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:00,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183167 s, 55.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:01,881::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:01,882::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70501::DEBUG::2014-09-25 17:31:01,884::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:02,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:02,748::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175038 s, 58.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:02,832::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:02,834::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70502::DEBUG::2014-09-25 17:31:02,834::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70502::DEBUG::2014-09-25 17:31:02,836::task::595::Storage.TaskManager.Task::(_updateState) Task=`e5ae8bed-fdfc-41af-9ca5-42df1b896bbe`::moving from state init -> state preparing >Thread-70502::INFO::2014-09-25 17:31:02,836::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70502::INFO::2014-09-25 17:31:02,837::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70502::DEBUG::2014-09-25 17:31:02,837::task::1191::Storage.TaskManager.Task::(prepare) Task=`e5ae8bed-fdfc-41af-9ca5-42df1b896bbe`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70502::DEBUG::2014-09-25 17:31:02,838::task::595::Storage.TaskManager.Task::(_updateState) Task=`e5ae8bed-fdfc-41af-9ca5-42df1b896bbe`::moving from state preparing -> state finished >Thread-70502::DEBUG::2014-09-25 17:31:02,838::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70502::DEBUG::2014-09-25 17:31:02,838::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70502::DEBUG::2014-09-25 17:31:02,838::task::993::Storage.TaskManager.Task::(_decref) Task=`e5ae8bed-fdfc-41af-9ca5-42df1b896bbe`::ref 0 aborting False >Thread-70502::DEBUG::2014-09-25 17:31:02,838::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70502::DEBUG::2014-09-25 17:31:02,839::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:02,849::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70503::DEBUG::2014-09-25 17:31:02,850::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:31:02,850::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70503::DEBUG::2014-09-25 17:31:02,852::task::595::Storage.TaskManager.Task::(_updateState) Task=`e22f1528-716e-4851-b022-99cafc7ed4c9`::moving from state init -> state preparing >Thread-70503::INFO::2014-09-25 17:31:02,853::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70503::DEBUG::2014-09-25 17:31:02,853::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`bffa53c2-914a-4c0c-ab1c-d2a4864e84a6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70503::DEBUG::2014-09-25 17:31:02,854::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70503::DEBUG::2014-09-25 17:31:02,854::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70503::DEBUG::2014-09-25 17:31:02,854::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`bffa53c2-914a-4c0c-ab1c-d2a4864e84a6`::Granted request >Thread-70503::DEBUG::2014-09-25 17:31:02,855::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e22f1528-716e-4851-b022-99cafc7ed4c9`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70503::DEBUG::2014-09-25 17:31:02,855::task::993::Storage.TaskManager.Task::(_decref) Task=`e22f1528-716e-4851-b022-99cafc7ed4c9`::ref 1 aborting False >Thread-70503::INFO::2014-09-25 17:31:02,859::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70503::DEBUG::2014-09-25 17:31:02,860::task::1191::Storage.TaskManager.Task::(prepare) Task=`e22f1528-716e-4851-b022-99cafc7ed4c9`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70503::DEBUG::2014-09-25 17:31:02,860::task::595::Storage.TaskManager.Task::(_updateState) Task=`e22f1528-716e-4851-b022-99cafc7ed4c9`::moving from state preparing -> state finished >Thread-70503::DEBUG::2014-09-25 17:31:02,860::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70503::DEBUG::2014-09-25 17:31:02,860::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70503::DEBUG::2014-09-25 17:31:02,861::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70503::DEBUG::2014-09-25 17:31:02,861::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70503::DEBUG::2014-09-25 17:31:02,861::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70503::DEBUG::2014-09-25 17:31:02,862::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70503::DEBUG::2014-09-25 17:31:02,862::task::993::Storage.TaskManager.Task::(_decref) Task=`e22f1528-716e-4851-b022-99cafc7ed4c9`::ref 0 aborting False >Thread-70503::DEBUG::2014-09-25 17:31:02,862::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70503::DEBUG::2014-09-25 17:31:02,863::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:31:04,457::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:04,458::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:04,458::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:04,458::__init__::232::IOProcess::(_processLogs) (108461) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:04,459::__init__::232::IOProcess::(_processLogs) (108461) Queuing response >Thread-80::DEBUG::2014-09-25 17:31:04,460::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:31:04,469::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000394699 s, 958 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:04,470::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:04,471::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:04,471::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:04,472::__init__::232::IOProcess::(_processLogs) (108462) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:04,473::__init__::232::IOProcess::(_processLogs) (108462) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:04,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:04,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172501 s, 59.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:04,897::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:04,899::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70504::DEBUG::2014-09-25 17:31:04,901::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:31:05,562::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:05,563::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:05,563::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:05,563::__init__::232::IOProcess::(_processLogs) (108463) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:05,569::__init__::232::IOProcess::(_processLogs) (108463) Queuing response >Thread-79::DEBUG::2014-09-25 17:31:05,570::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:31:05,595::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000600369 s, 936 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:05,596::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:05,597::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:05,597::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:05,597::__init__::232::IOProcess::(_processLogs) (108464) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:05,611::__init__::232::IOProcess::(_processLogs) (108464) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:05,613::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:05,613::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:05,614::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:05,614::__init__::232::IOProcess::(_processLogs) (108465) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:05,614::__init__::232::IOProcess::(_processLogs) (108465) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:05,615::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:05,615::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:05,616::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:05,616::__init__::232::IOProcess::(_processLogs) (108466) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:05,617::__init__::232::IOProcess::(_processLogs) (108466) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:05,617::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:05,618::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:05,619::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:05,619::__init__::232::IOProcess::(_processLogs) (108467) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:05,619::__init__::232::IOProcess::(_processLogs) (108467) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:06,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:06,847::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202132 s, 50.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:07,920::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:07,921::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70505::DEBUG::2014-09-25 17:31:07,923::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:08,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:08,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144584 s, 70.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:10,651::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:10,652::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:10,652::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:10,652::__init__::232::IOProcess::(_processLogs) (108468) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:10,655::__init__::232::IOProcess::(_processLogs) (108468) Queuing response >Thread-81::DEBUG::2014-09-25 17:31:10,656::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:31:10,680::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00849436 s, 43.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:10,681::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:10,681::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:10,682::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:10,682::__init__::232::IOProcess::(_processLogs) (108469) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:10,688::__init__::232::IOProcess::(_processLogs) (108469) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:10,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:10,935::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:10,938::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70506::DEBUG::2014-09-25 17:31:10,944::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:10,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0271065 s, 37.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:12,889::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:12,891::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70507::DEBUG::2014-09-25 17:31:12,891::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70507::DEBUG::2014-09-25 17:31:12,893::task::595::Storage.TaskManager.Task::(_updateState) Task=`7c76e4ae-6023-4b90-9b58-829f98a4547b`::moving from state init -> state preparing >Thread-70507::INFO::2014-09-25 17:31:12,893::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70507::INFO::2014-09-25 17:31:12,894::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70507::DEBUG::2014-09-25 17:31:12,894::task::1191::Storage.TaskManager.Task::(prepare) Task=`7c76e4ae-6023-4b90-9b58-829f98a4547b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70507::DEBUG::2014-09-25 17:31:12,894::task::595::Storage.TaskManager.Task::(_updateState) Task=`7c76e4ae-6023-4b90-9b58-829f98a4547b`::moving from state preparing -> state finished >Thread-70507::DEBUG::2014-09-25 17:31:12,894::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70507::DEBUG::2014-09-25 17:31:12,894::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70507::DEBUG::2014-09-25 17:31:12,895::task::993::Storage.TaskManager.Task::(_decref) Task=`7c76e4ae-6023-4b90-9b58-829f98a4547b`::ref 0 aborting False >Thread-70507::DEBUG::2014-09-25 17:31:12,895::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70507::DEBUG::2014-09-25 17:31:12,895::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:12,910::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:12,910::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70508::DEBUG::2014-09-25 17:31:12,911::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70508::DEBUG::2014-09-25 17:31:12,912::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a29cefc-960f-4af5-9e30-c44e20637166`::moving from state init -> state preparing >Thread-70508::INFO::2014-09-25 17:31:12,913::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70508::DEBUG::2014-09-25 17:31:12,913::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c8039a2a-4d1f-4a92-b32e-38a6c100c8a2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70508::DEBUG::2014-09-25 17:31:12,914::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70508::DEBUG::2014-09-25 17:31:12,914::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70508::DEBUG::2014-09-25 17:31:12,914::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c8039a2a-4d1f-4a92-b32e-38a6c100c8a2`::Granted request >Thread-70508::DEBUG::2014-09-25 17:31:12,915::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0a29cefc-960f-4af5-9e30-c44e20637166`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70508::DEBUG::2014-09-25 17:31:12,915::task::993::Storage.TaskManager.Task::(_decref) Task=`0a29cefc-960f-4af5-9e30-c44e20637166`::ref 1 aborting False >Thread-70508::INFO::2014-09-25 17:31:12,919::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70508::DEBUG::2014-09-25 17:31:12,920::task::1191::Storage.TaskManager.Task::(prepare) Task=`0a29cefc-960f-4af5-9e30-c44e20637166`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70508::DEBUG::2014-09-25 17:31:12,920::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a29cefc-960f-4af5-9e30-c44e20637166`::moving from state preparing -> state finished >Thread-70508::DEBUG::2014-09-25 17:31:12,920::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70508::DEBUG::2014-09-25 17:31:12,920::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70508::DEBUG::2014-09-25 17:31:12,921::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70508::DEBUG::2014-09-25 17:31:12,921::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70508::DEBUG::2014-09-25 17:31:12,921::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70508::DEBUG::2014-09-25 17:31:12,922::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70508::DEBUG::2014-09-25 17:31:12,922::task::993::Storage.TaskManager.Task::(_decref) Task=`0a29cefc-960f-4af5-9e30-c44e20637166`::ref 0 aborting False >Thread-70508::DEBUG::2014-09-25 17:31:12,922::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70508::DEBUG::2014-09-25 17:31:12,923::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:12,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:13,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.239886 s, 4.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:13,962::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:13,964::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70509::DEBUG::2014-09-25 17:31:13,966::task::595::Storage.TaskManager.Task::(_updateState) Task=`34fc7504-d363-4d63-9789-d000899099ed`::moving from state init -> state preparing >Thread-70509::INFO::2014-09-25 17:31:13,966::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70509::INFO::2014-09-25 17:31:13,967::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000600369', 'lastCheck': '8.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000394699', 'lastCheck': '9.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00849436', 'lastCheck': '3.3', 'valid': True}} >Thread-70509::DEBUG::2014-09-25 17:31:13,967::task::1191::Storage.TaskManager.Task::(prepare) Task=`34fc7504-d363-4d63-9789-d000899099ed`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000600369', 'lastCheck': '8.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000394699', 'lastCheck': '9.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00849436', 'lastCheck': '3.3', 'valid': True}} >Thread-70509::DEBUG::2014-09-25 17:31:13,967::task::595::Storage.TaskManager.Task::(_updateState) Task=`34fc7504-d363-4d63-9789-d000899099ed`::moving from state preparing -> state finished >Thread-70509::DEBUG::2014-09-25 17:31:13,968::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70509::DEBUG::2014-09-25 17:31:13,968::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70509::DEBUG::2014-09-25 17:31:13,968::task::993::Storage.TaskManager.Task::(_decref) Task=`34fc7504-d363-4d63-9789-d000899099ed`::ref 0 aborting False >Thread-70509::DEBUG::2014-09-25 17:31:13,970::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:13,992::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:13,994::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70510::DEBUG::2014-09-25 17:31:13,996::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:31:14,474::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:14,474::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:14,475::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:14,476::__init__::232::IOProcess::(_processLogs) (108470) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:31:14,476::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:31:14,477::__init__::232::IOProcess::(_processLogs) (108470) Queuing response >Thread-80::DEBUG::2014-09-25 17:31:14,487::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000619444 s, 610 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:14,488::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:14,488::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:14,488::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:14,490::__init__::232::IOProcess::(_processLogs) (108471) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:14,490::__init__::232::IOProcess::(_processLogs) (108471) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:15,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:15,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0221528 s, 46.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:15,619::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:15,620::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:15,620::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:15,620::__init__::232::IOProcess::(_processLogs) (108472) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:15,626::__init__::232::IOProcess::(_processLogs) (108472) Queuing response >Thread-79::DEBUG::2014-09-25 17:31:15,627::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:31:15,644::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00247574 s, 227 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:15,645::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:15,646::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:15,646::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:15,646::__init__::232::IOProcess::(_processLogs) (108473) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:15,651::__init__::232::IOProcess::(_processLogs) (108473) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:15,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:15,653::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:15,653::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:15,654::__init__::232::IOProcess::(_processLogs) (108474) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:15,654::__init__::232::IOProcess::(_processLogs) (108474) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:15,655::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:15,655::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:15,655::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:15,656::__init__::232::IOProcess::(_processLogs) (108475) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:15,656::__init__::232::IOProcess::(_processLogs) (108475) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:15,657::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:15,657::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:15,658::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:15,658::__init__::232::IOProcess::(_processLogs) (108476) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:15,659::__init__::232::IOProcess::(_processLogs) (108476) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:17,055::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:17,058::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70511::DEBUG::2014-09-25 17:31:17,063::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:17,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:17,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193886 s, 52.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:31:19,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:19,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133026 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:20,084::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:20,086::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70512::DEBUG::2014-09-25 17:31:20,088::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:31:20,690::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:20,690::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:20,690::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:20,690::__init__::232::IOProcess::(_processLogs) (108477) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:20,696::__init__::232::IOProcess::(_processLogs) (108477) Queuing response >Thread-81::DEBUG::2014-09-25 17:31:20,696::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:31:20,708::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000383935 s, 966 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:20,709::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:20,709::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:20,710::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:20,710::__init__::232::IOProcess::(_processLogs) (108478) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:20,710::__init__::232::IOProcess::(_processLogs) (108478) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:21,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:21,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170346 s, 60.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:22,956::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70513::DEBUG::2014-09-25 17:31:22,957::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:31:22,957::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70513::DEBUG::2014-09-25 17:31:22,959::task::595::Storage.TaskManager.Task::(_updateState) Task=`ae664bcc-8f11-43e4-841e-c2a2c5ba10a8`::moving from state init -> state preparing >Thread-70513::INFO::2014-09-25 17:31:22,960::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70513::INFO::2014-09-25 17:31:22,960::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70513::DEBUG::2014-09-25 17:31:22,961::task::1191::Storage.TaskManager.Task::(prepare) Task=`ae664bcc-8f11-43e4-841e-c2a2c5ba10a8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70513::DEBUG::2014-09-25 17:31:22,961::task::595::Storage.TaskManager.Task::(_updateState) Task=`ae664bcc-8f11-43e4-841e-c2a2c5ba10a8`::moving from state preparing -> state finished >Thread-70513::DEBUG::2014-09-25 17:31:22,961::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70513::DEBUG::2014-09-25 17:31:22,961::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70513::DEBUG::2014-09-25 17:31:22,961::task::993::Storage.TaskManager.Task::(_decref) Task=`ae664bcc-8f11-43e4-841e-c2a2c5ba10a8`::ref 0 aborting False >Thread-70513::DEBUG::2014-09-25 17:31:22,962::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70513::DEBUG::2014-09-25 17:31:22,962::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:22,981::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70514::DEBUG::2014-09-25 17:31:22,982::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:31:22,982::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70514::DEBUG::2014-09-25 17:31:22,985::task::595::Storage.TaskManager.Task::(_updateState) Task=`48159b66-52e6-4f2f-9c8d-8d3d6e4c373d`::moving from state init -> state preparing >Thread-70514::INFO::2014-09-25 17:31:22,985::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70514::DEBUG::2014-09-25 17:31:22,986::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`391b76a7-0c74-4e30-b203-1d98ae997c69`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70514::DEBUG::2014-09-25 17:31:22,986::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70514::DEBUG::2014-09-25 17:31:22,986::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70514::DEBUG::2014-09-25 17:31:22,986::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`391b76a7-0c74-4e30-b203-1d98ae997c69`::Granted request >Thread-70514::DEBUG::2014-09-25 17:31:22,987::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`48159b66-52e6-4f2f-9c8d-8d3d6e4c373d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70514::DEBUG::2014-09-25 17:31:22,987::task::993::Storage.TaskManager.Task::(_decref) Task=`48159b66-52e6-4f2f-9c8d-8d3d6e4c373d`::ref 1 aborting False >Thread-70514::INFO::2014-09-25 17:31:22,992::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70514::DEBUG::2014-09-25 17:31:22,992::task::1191::Storage.TaskManager.Task::(prepare) Task=`48159b66-52e6-4f2f-9c8d-8d3d6e4c373d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70514::DEBUG::2014-09-25 17:31:22,992::task::595::Storage.TaskManager.Task::(_updateState) Task=`48159b66-52e6-4f2f-9c8d-8d3d6e4c373d`::moving from state preparing -> state finished >Thread-70514::DEBUG::2014-09-25 17:31:22,992::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70514::DEBUG::2014-09-25 17:31:22,993::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70514::DEBUG::2014-09-25 17:31:22,993::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70514::DEBUG::2014-09-25 17:31:22,993::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70514::DEBUG::2014-09-25 17:31:22,994::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70514::DEBUG::2014-09-25 17:31:22,994::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70514::DEBUG::2014-09-25 17:31:22,994::task::993::Storage.TaskManager.Task::(_decref) Task=`48159b66-52e6-4f2f-9c8d-8d3d6e4c373d`::ref 0 aborting False >Thread-70514::DEBUG::2014-09-25 17:31:22,995::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70514::DEBUG::2014-09-25 17:31:22,995::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:23,106::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:23,107::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70515::DEBUG::2014-09-25 17:31:23,109::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:23,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:23,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131777 s, 77.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:24,491::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:24,492::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:24,492::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:24,492::__init__::232::IOProcess::(_processLogs) (108479) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:24,492::__init__::232::IOProcess::(_processLogs) (108479) Queuing response >Thread-80::DEBUG::2014-09-25 17:31:24,494::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:31:24,503::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00054464 s, 694 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:24,504::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:24,504::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:24,504::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:24,504::__init__::232::IOProcess::(_processLogs) (108480) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:24,505::__init__::232::IOProcess::(_processLogs) (108480) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:25,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:25,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165044 s, 62.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:25,659::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:25,659::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:25,660::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:25,660::__init__::232::IOProcess::(_processLogs) (108481) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:25,660::__init__::232::IOProcess::(_processLogs) (108481) Queuing response >Thread-79::DEBUG::2014-09-25 17:31:25,661::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:31:25,670::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000525653 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:25,672::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:25,672::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:25,672::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:25,672::__init__::232::IOProcess::(_processLogs) (108482) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:25,673::__init__::232::IOProcess::(_processLogs) (108482) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:25,674::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:25,675::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:25,675::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:25,676::__init__::232::IOProcess::(_processLogs) (108483) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:25,676::__init__::232::IOProcess::(_processLogs) (108483) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:25,677::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:25,677::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:25,677::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:25,678::__init__::232::IOProcess::(_processLogs) (108484) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:25,678::__init__::232::IOProcess::(_processLogs) (108484) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:25,679::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:25,679::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:25,680::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:25,680::__init__::232::IOProcess::(_processLogs) (108485) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:25,681::__init__::232::IOProcess::(_processLogs) (108485) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:26,127::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:26,128::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70516::DEBUG::2014-09-25 17:31:26,130::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:27,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:27,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191233 s, 53.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:29,143::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:29,144::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70517::DEBUG::2014-09-25 17:31:29,147::task::595::Storage.TaskManager.Task::(_updateState) Task=`f26e93e3-964d-44c5-87c4-48f5a91db4bb`::moving from state init -> state preparing >Thread-70517::INFO::2014-09-25 17:31:29,147::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70517::INFO::2014-09-25 17:31:29,148::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000525653', 'lastCheck': '3.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00054464', 'lastCheck': '4.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000383935', 'lastCheck': '8.4', 'valid': True}} >Thread-70517::DEBUG::2014-09-25 17:31:29,148::task::1191::Storage.TaskManager.Task::(prepare) Task=`f26e93e3-964d-44c5-87c4-48f5a91db4bb`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000525653', 'lastCheck': '3.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00054464', 'lastCheck': '4.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000383935', 'lastCheck': '8.4', 'valid': True}} >Thread-70517::DEBUG::2014-09-25 17:31:29,148::task::595::Storage.TaskManager.Task::(_updateState) Task=`f26e93e3-964d-44c5-87c4-48f5a91db4bb`::moving from state preparing -> state finished >Thread-70517::DEBUG::2014-09-25 17:31:29,148::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70517::DEBUG::2014-09-25 17:31:29,149::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70517::DEBUG::2014-09-25 17:31:29,149::task::993::Storage.TaskManager.Task::(_decref) Task=`f26e93e3-964d-44c5-87c4-48f5a91db4bb`::ref 0 aborting False >Thread-70517::DEBUG::2014-09-25 17:31:29,151::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:29,168::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:29,170::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70518::DEBUG::2014-09-25 17:31:29,172::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:29,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:29,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155783 s, 65.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:30,713::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:30,713::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:30,713::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:30,714::__init__::232::IOProcess::(_processLogs) (108486) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:30,716::__init__::232::IOProcess::(_processLogs) (108486) Queuing response >Thread-81::DEBUG::2014-09-25 17:31:30,716::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:31:30,733::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00408954 s, 90.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:30,734::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:30,735::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:30,735::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:30,735::__init__::232::IOProcess::(_processLogs) (108487) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:30,738::__init__::232::IOProcess::(_processLogs) (108487) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:31,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:31,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183786 s, 55.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:32,222::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:32,224::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70519::DEBUG::2014-09-25 17:31:32,226::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:33,032::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:33,034::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70520::DEBUG::2014-09-25 17:31:33,034::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70520::DEBUG::2014-09-25 17:31:33,036::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ccf4ab0-29fb-41d2-9d3d-34b684add4c6`::moving from state init -> state preparing >Thread-70520::INFO::2014-09-25 17:31:33,036::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70520::INFO::2014-09-25 17:31:33,037::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70520::DEBUG::2014-09-25 17:31:33,037::task::1191::Storage.TaskManager.Task::(prepare) Task=`8ccf4ab0-29fb-41d2-9d3d-34b684add4c6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70520::DEBUG::2014-09-25 17:31:33,037::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ccf4ab0-29fb-41d2-9d3d-34b684add4c6`::moving from state preparing -> state finished >Thread-70520::DEBUG::2014-09-25 17:31:33,037::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70520::DEBUG::2014-09-25 17:31:33,038::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70520::DEBUG::2014-09-25 17:31:33,038::task::993::Storage.TaskManager.Task::(_decref) Task=`8ccf4ab0-29fb-41d2-9d3d-34b684add4c6`::ref 0 aborting False >Thread-70520::DEBUG::2014-09-25 17:31:33,038::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70520::DEBUG::2014-09-25 17:31:33,038::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:33,049::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:33,050::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70521::DEBUG::2014-09-25 17:31:33,050::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70521::DEBUG::2014-09-25 17:31:33,052::task::595::Storage.TaskManager.Task::(_updateState) Task=`242504cc-06b1-44a3-b124-36ac799fabfb`::moving from state init -> state preparing >Thread-70521::INFO::2014-09-25 17:31:33,052::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70521::DEBUG::2014-09-25 17:31:33,054::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`757d804b-5a5c-4128-9c14-ab7eda77779d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70521::DEBUG::2014-09-25 17:31:33,054::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70521::DEBUG::2014-09-25 17:31:33,055::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70521::DEBUG::2014-09-25 17:31:33,055::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`757d804b-5a5c-4128-9c14-ab7eda77779d`::Granted request >Thread-70521::DEBUG::2014-09-25 17:31:33,056::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`242504cc-06b1-44a3-b124-36ac799fabfb`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70521::DEBUG::2014-09-25 17:31:33,056::task::993::Storage.TaskManager.Task::(_decref) Task=`242504cc-06b1-44a3-b124-36ac799fabfb`::ref 1 aborting False >Thread-70521::INFO::2014-09-25 17:31:33,060::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70521::DEBUG::2014-09-25 17:31:33,060::task::1191::Storage.TaskManager.Task::(prepare) Task=`242504cc-06b1-44a3-b124-36ac799fabfb`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70521::DEBUG::2014-09-25 17:31:33,061::task::595::Storage.TaskManager.Task::(_updateState) Task=`242504cc-06b1-44a3-b124-36ac799fabfb`::moving from state preparing -> state finished >Thread-70521::DEBUG::2014-09-25 17:31:33,061::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70521::DEBUG::2014-09-25 17:31:33,061::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70521::DEBUG::2014-09-25 17:31:33,061::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70521::DEBUG::2014-09-25 17:31:33,062::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70521::DEBUG::2014-09-25 17:31:33,062::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70521::DEBUG::2014-09-25 17:31:33,062::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70521::DEBUG::2014-09-25 17:31:33,063::task::993::Storage.TaskManager.Task::(_decref) Task=`242504cc-06b1-44a3-b124-36ac799fabfb`::ref 0 aborting False >Thread-70521::DEBUG::2014-09-25 17:31:33,063::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70521::DEBUG::2014-09-25 17:31:33,064::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:33,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:33,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192029 s, 53.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:34,507::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:34,508::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:34,508::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:34,508::__init__::232::IOProcess::(_processLogs) (108488) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:34,508::__init__::232::IOProcess::(_processLogs) (108488) Queuing response >Thread-80::DEBUG::2014-09-25 17:31:34,510::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:31:34,518::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.0004958 s, 762 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:34,520::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:34,520::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:34,520::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:34,520::__init__::232::IOProcess::(_processLogs) (108489) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:34,521::__init__::232::IOProcess::(_processLogs) (108489) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:35,241::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:35,243::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70522::DEBUG::2014-09-25 17:31:35,245::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:31:35,681::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:35,681::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:35,682::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:35,682::__init__::232::IOProcess::(_processLogs) (108490) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:35,684::__init__::232::IOProcess::(_processLogs) (108490) Queuing response >Thread-79::DEBUG::2014-09-25 17:31:35,684::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:31:35,696::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00155944 s, 360 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:35,698::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:35,698::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:35,698::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:35,698::__init__::232::IOProcess::(_processLogs) (108491) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:35,699::__init__::232::IOProcess::(_processLogs) (108491) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:35,700::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:35,701::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:35,701::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:35,701::__init__::232::IOProcess::(_processLogs) (108492) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:35,702::__init__::232::IOProcess::(_processLogs) (108492) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:35,702::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:35,703::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:35,703::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:35,703::__init__::232::IOProcess::(_processLogs) (108493) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:35,704::__init__::232::IOProcess::(_processLogs) (108493) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:35,704::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:35,704::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:35,705::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:35,706::__init__::232::IOProcess::(_processLogs) (108494) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:35,706::__init__::232::IOProcess::(_processLogs) (108494) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:35,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:35,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168786 s, 60.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:31:37,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:37,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018567 s, 55.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:38,260::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:38,261::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70523::DEBUG::2014-09-25 17:31:38,263::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:39,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:39,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199155 s, 51.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:40,741::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:40,741::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:40,741::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:40,741::__init__::232::IOProcess::(_processLogs) (108495) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:40,743::__init__::232::IOProcess::(_processLogs) (108495) Queuing response >Thread-81::DEBUG::2014-09-25 17:31:40,744::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:31:40,755::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00158175 s, 235 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:40,757::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:40,757::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:40,757::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:40,757::__init__::232::IOProcess::(_processLogs) (108496) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:40,760::__init__::232::IOProcess::(_processLogs) (108496) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:41,277::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:41,279::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70524::DEBUG::2014-09-25 17:31:41,281::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:41,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:41,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0197221 s, 51.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:43,087::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70525::DEBUG::2014-09-25 17:31:43,088::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:31:43,088::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70525::DEBUG::2014-09-25 17:31:43,091::task::595::Storage.TaskManager.Task::(_updateState) Task=`4672392f-acd6-482b-970d-08aee95d2937`::moving from state init -> state preparing >Thread-70525::INFO::2014-09-25 17:31:43,091::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70525::INFO::2014-09-25 17:31:43,091::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70525::DEBUG::2014-09-25 17:31:43,092::task::1191::Storage.TaskManager.Task::(prepare) Task=`4672392f-acd6-482b-970d-08aee95d2937`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70525::DEBUG::2014-09-25 17:31:43,092::task::595::Storage.TaskManager.Task::(_updateState) Task=`4672392f-acd6-482b-970d-08aee95d2937`::moving from state preparing -> state finished >Thread-70525::DEBUG::2014-09-25 17:31:43,092::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70525::DEBUG::2014-09-25 17:31:43,092::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70525::DEBUG::2014-09-25 17:31:43,093::task::993::Storage.TaskManager.Task::(_decref) Task=`4672392f-acd6-482b-970d-08aee95d2937`::ref 0 aborting False >Thread-70525::DEBUG::2014-09-25 17:31:43,093::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70525::DEBUG::2014-09-25 17:31:43,093::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:43,107::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:43,109::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70526::DEBUG::2014-09-25 17:31:43,109::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70526::DEBUG::2014-09-25 17:31:43,110::task::595::Storage.TaskManager.Task::(_updateState) Task=`546c6b77-cad2-41f9-a49c-89e236501b69`::moving from state init -> state preparing >Thread-70526::INFO::2014-09-25 17:31:43,111::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70526::DEBUG::2014-09-25 17:31:43,111::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`60a2050a-a424-4060-84cf-f1bf40ed5475`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70526::DEBUG::2014-09-25 17:31:43,112::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70526::DEBUG::2014-09-25 17:31:43,112::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70526::DEBUG::2014-09-25 17:31:43,112::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`60a2050a-a424-4060-84cf-f1bf40ed5475`::Granted request >Thread-70526::DEBUG::2014-09-25 17:31:43,113::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`546c6b77-cad2-41f9-a49c-89e236501b69`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70526::DEBUG::2014-09-25 17:31:43,113::task::993::Storage.TaskManager.Task::(_decref) Task=`546c6b77-cad2-41f9-a49c-89e236501b69`::ref 1 aborting False >Thread-70526::INFO::2014-09-25 17:31:43,118::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70526::DEBUG::2014-09-25 17:31:43,118::task::1191::Storage.TaskManager.Task::(prepare) Task=`546c6b77-cad2-41f9-a49c-89e236501b69`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70526::DEBUG::2014-09-25 17:31:43,118::task::595::Storage.TaskManager.Task::(_updateState) Task=`546c6b77-cad2-41f9-a49c-89e236501b69`::moving from state preparing -> state finished >Thread-70526::DEBUG::2014-09-25 17:31:43,119::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70526::DEBUG::2014-09-25 17:31:43,119::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70526::DEBUG::2014-09-25 17:31:43,119::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70526::DEBUG::2014-09-25 17:31:43,119::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70526::DEBUG::2014-09-25 17:31:43,120::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70526::DEBUG::2014-09-25 17:31:43,120::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70526::DEBUG::2014-09-25 17:31:43,120::task::993::Storage.TaskManager.Task::(_decref) Task=`546c6b77-cad2-41f9-a49c-89e236501b69`::ref 0 aborting False >Thread-70526::DEBUG::2014-09-25 17:31:43,121::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70526::DEBUG::2014-09-25 17:31:43,121::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:43,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:43,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133256 s, 76.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:44,300::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:44,302::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70527::DEBUG::2014-09-25 17:31:44,305::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a845186-be33-4c59-b9f8-3aede724fc4b`::moving from state init -> state preparing >Thread-70527::INFO::2014-09-25 17:31:44,305::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70527::INFO::2014-09-25 17:31:44,305::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00155944', 'lastCheck': '8.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0004958', 'lastCheck': '9.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00158175', 'lastCheck': '3.5', 'valid': True}} >Thread-70527::DEBUG::2014-09-25 17:31:44,306::task::1191::Storage.TaskManager.Task::(prepare) Task=`0a845186-be33-4c59-b9f8-3aede724fc4b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00155944', 'lastCheck': '8.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0004958', 'lastCheck': '9.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00158175', 'lastCheck': '3.5', 'valid': True}} >Thread-70527::DEBUG::2014-09-25 17:31:44,306::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a845186-be33-4c59-b9f8-3aede724fc4b`::moving from state preparing -> state finished >Thread-70527::DEBUG::2014-09-25 17:31:44,306::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70527::DEBUG::2014-09-25 17:31:44,306::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70527::DEBUG::2014-09-25 17:31:44,307::task::993::Storage.TaskManager.Task::(_decref) Task=`0a845186-be33-4c59-b9f8-3aede724fc4b`::ref 0 aborting False >Thread-70527::DEBUG::2014-09-25 17:31:44,309::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:44,337::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:44,338::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70528::DEBUG::2014-09-25 17:31:44,340::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:31:44,523::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:44,524::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:44,524::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:44,524::__init__::232::IOProcess::(_processLogs) (108497) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:44,525::__init__::232::IOProcess::(_processLogs) (108497) Queuing response >Thread-80::DEBUG::2014-09-25 17:31:44,526::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:31:44,537::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000407868 s, 927 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:44,540::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:44,540::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:44,540::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:44,541::__init__::232::IOProcess::(_processLogs) (108498) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:44,542::__init__::232::IOProcess::(_processLogs) (108498) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:45,707::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:45,707::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:45,707::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:45,707::__init__::232::IOProcess::(_processLogs) (108499) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:45,708::__init__::232::IOProcess::(_processLogs) (108499) Queuing response >Thread-79::DEBUG::2014-09-25 17:31:45,709::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:31:45,718::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000452645 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:45,719::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:45,720::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:45,720::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:45,720::__init__::232::IOProcess::(_processLogs) (108500) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:45,721::__init__::232::IOProcess::(_processLogs) (108500) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:45,722::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:45,722::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:45,723::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:45,723::__init__::232::IOProcess::(_processLogs) (108501) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:45,724::__init__::232::IOProcess::(_processLogs) (108501) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:45,724::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:45,725::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:45,725::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:45,725::__init__::232::IOProcess::(_processLogs) (108502) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:45,726::__init__::232::IOProcess::(_processLogs) (108502) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:45,727::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:45,727::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:45,727::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:45,728::__init__::232::IOProcess::(_processLogs) (108503) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:45,729::__init__::232::IOProcess::(_processLogs) (108503) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:45,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:46,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0244031 s, 42.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:47,398::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:47,399::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70529::DEBUG::2014-09-25 17:31:47,400::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:48,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:48,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165097 s, 62.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:31:50,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:50,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0246216 s, 41.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:50,413::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:50,415::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70530::DEBUG::2014-09-25 17:31:50,417::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:31:50,761::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:50,762::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:50,762::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:50,762::__init__::232::IOProcess::(_processLogs) (108504) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:50,766::__init__::232::IOProcess::(_processLogs) (108504) Queuing response >Thread-81::DEBUG::2014-09-25 17:31:50,766::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:31:50,782::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0067636 s, 54.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:50,783::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:50,783::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:50,784::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:50,784::__init__::232::IOProcess::(_processLogs) (108505) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:50,785::__init__::232::IOProcess::(_processLogs) (108505) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:52,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:52,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177076 s, 57.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:53,151::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70531::DEBUG::2014-09-25 17:31:53,152::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:31:53,152::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70531::DEBUG::2014-09-25 17:31:53,155::task::595::Storage.TaskManager.Task::(_updateState) Task=`6db32dd1-9aa9-447d-9386-9da092664d6b`::moving from state init -> state preparing >Thread-70531::INFO::2014-09-25 17:31:53,155::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70531::INFO::2014-09-25 17:31:53,156::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70531::DEBUG::2014-09-25 17:31:53,156::task::1191::Storage.TaskManager.Task::(prepare) Task=`6db32dd1-9aa9-447d-9386-9da092664d6b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70531::DEBUG::2014-09-25 17:31:53,156::task::595::Storage.TaskManager.Task::(_updateState) Task=`6db32dd1-9aa9-447d-9386-9da092664d6b`::moving from state preparing -> state finished >Thread-70531::DEBUG::2014-09-25 17:31:53,156::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70531::DEBUG::2014-09-25 17:31:53,157::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70531::DEBUG::2014-09-25 17:31:53,157::task::993::Storage.TaskManager.Task::(_decref) Task=`6db32dd1-9aa9-447d-9386-9da092664d6b`::ref 0 aborting False >Thread-70531::DEBUG::2014-09-25 17:31:53,157::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70531::DEBUG::2014-09-25 17:31:53,158::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:53,171::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:53,172::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70532::DEBUG::2014-09-25 17:31:53,172::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70532::DEBUG::2014-09-25 17:31:53,174::task::595::Storage.TaskManager.Task::(_updateState) Task=`2476823e-5d46-4eab-b390-9ecac27e4743`::moving from state init -> state preparing >Thread-70532::INFO::2014-09-25 17:31:53,174::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70532::DEBUG::2014-09-25 17:31:53,175::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d01150ae-181e-4a65-bbfd-4ecf668e719f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70532::DEBUG::2014-09-25 17:31:53,175::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70532::DEBUG::2014-09-25 17:31:53,175::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70532::DEBUG::2014-09-25 17:31:53,176::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d01150ae-181e-4a65-bbfd-4ecf668e719f`::Granted request >Thread-70532::DEBUG::2014-09-25 17:31:53,176::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2476823e-5d46-4eab-b390-9ecac27e4743`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70532::DEBUG::2014-09-25 17:31:53,176::task::993::Storage.TaskManager.Task::(_decref) Task=`2476823e-5d46-4eab-b390-9ecac27e4743`::ref 1 aborting False >Thread-70532::INFO::2014-09-25 17:31:53,181::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70532::DEBUG::2014-09-25 17:31:53,181::task::1191::Storage.TaskManager.Task::(prepare) Task=`2476823e-5d46-4eab-b390-9ecac27e4743`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70532::DEBUG::2014-09-25 17:31:53,181::task::595::Storage.TaskManager.Task::(_updateState) Task=`2476823e-5d46-4eab-b390-9ecac27e4743`::moving from state preparing -> state finished >Thread-70532::DEBUG::2014-09-25 17:31:53,182::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70532::DEBUG::2014-09-25 17:31:53,182::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70532::DEBUG::2014-09-25 17:31:53,182::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70532::DEBUG::2014-09-25 17:31:53,182::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70532::DEBUG::2014-09-25 17:31:53,183::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70532::DEBUG::2014-09-25 17:31:53,183::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70532::DEBUG::2014-09-25 17:31:53,183::task::993::Storage.TaskManager.Task::(_decref) Task=`2476823e-5d46-4eab-b390-9ecac27e4743`::ref 0 aborting False >Thread-70532::DEBUG::2014-09-25 17:31:53,184::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70532::DEBUG::2014-09-25 17:31:53,184::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:53,433::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:53,434::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70533::DEBUG::2014-09-25 17:31:53,436::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:54,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:54,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202315 s, 50.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:54,544::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:54,544::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:54,544::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:54,545::__init__::232::IOProcess::(_processLogs) (108506) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:54,545::__init__::232::IOProcess::(_processLogs) (108506) Queuing response >Thread-80::DEBUG::2014-09-25 17:31:54,546::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:31:54,556::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000528923 s, 715 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:54,557::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:54,557::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:54,557::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:54,558::__init__::232::IOProcess::(_processLogs) (108507) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:54,558::__init__::232::IOProcess::(_processLogs) (108507) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:55,729::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:55,730::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:55,730::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:55,730::__init__::232::IOProcess::(_processLogs) (108508) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:55,735::__init__::232::IOProcess::(_processLogs) (108508) Queuing response >Thread-79::DEBUG::2014-09-25 17:31:55,736::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:31:55,766::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00180609 s, 311 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:31:55,767::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:55,768::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:55,768::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:55,768::__init__::232::IOProcess::(_processLogs) (108509) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:31:55,773::__init__::232::IOProcess::(_processLogs) (108509) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:55,774::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:55,775::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:55,775::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:55,776::__init__::232::IOProcess::(_processLogs) (108510) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:55,776::__init__::232::IOProcess::(_processLogs) (108510) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:55,776::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:55,777::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:55,778::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:55,778::__init__::232::IOProcess::(_processLogs) (108511) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:55,778::__init__::232::IOProcess::(_processLogs) (108511) Queuing response >Thread-29::DEBUG::2014-09-25 17:31:55,778::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:31:55,779::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:31:55,780::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:31:55,780::__init__::232::IOProcess::(_processLogs) (108512) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:31:55,780::__init__::232::IOProcess::(_processLogs) (108512) Queuing response >Dummy-148::DEBUG::2014-09-25 17:31:56,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:56,450::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:56,452::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70534::DEBUG::2014-09-25 17:31:56,454::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:31:56,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131713 s, 77.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:31:58,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:31:58,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0250476 s, 40.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:59,466::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:59,467::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70535::DEBUG::2014-09-25 17:31:59,470::task::595::Storage.TaskManager.Task::(_updateState) Task=`769869fa-ae35-46a2-af10-3063fec42ed6`::moving from state init -> state preparing >Thread-70535::INFO::2014-09-25 17:31:59,470::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70535::INFO::2014-09-25 17:31:59,471::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00180609', 'lastCheck': '3.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000528923', 'lastCheck': '4.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0067636', 'lastCheck': '8.7', 'valid': True}} >Thread-70535::DEBUG::2014-09-25 17:31:59,471::task::1191::Storage.TaskManager.Task::(prepare) Task=`769869fa-ae35-46a2-af10-3063fec42ed6`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00180609', 'lastCheck': '3.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000528923', 'lastCheck': '4.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0067636', 'lastCheck': '8.7', 'valid': True}} >Thread-70535::DEBUG::2014-09-25 17:31:59,471::task::595::Storage.TaskManager.Task::(_updateState) Task=`769869fa-ae35-46a2-af10-3063fec42ed6`::moving from state preparing -> state finished >Thread-70535::DEBUG::2014-09-25 17:31:59,471::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70535::DEBUG::2014-09-25 17:31:59,472::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70535::DEBUG::2014-09-25 17:31:59,472::task::993::Storage.TaskManager.Task::(_decref) Task=`769869fa-ae35-46a2-af10-3063fec42ed6`::ref 0 aborting False >Thread-70535::DEBUG::2014-09-25 17:31:59,474::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:31:59,489::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:31:59,491::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70536::DEBUG::2014-09-25 17:31:59,492::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:00,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:00,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131183 s, 78.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:00,787::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:00,788::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:00,788::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:00,788::__init__::232::IOProcess::(_processLogs) (108513) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:00,789::__init__::232::IOProcess::(_processLogs) (108513) Queuing response >Thread-81::DEBUG::2014-09-25 17:32:00,790::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:32:00,799::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000362216 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:00,800::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:00,801::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:00,801::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:00,803::__init__::232::IOProcess::(_processLogs) (108514) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:00,803::__init__::232::IOProcess::(_processLogs) (108514) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:02,551::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:02,552::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70537::DEBUG::2014-09-25 17:32:02,554::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:02,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:02,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.21165 s, 4.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:03,210::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70538::DEBUG::2014-09-25 17:32:03,212::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:32:03,212::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70538::DEBUG::2014-09-25 17:32:03,214::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ae1de82-3179-42a0-bfdd-e6b06d6a0362`::moving from state init -> state preparing >Thread-70538::INFO::2014-09-25 17:32:03,214::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70538::INFO::2014-09-25 17:32:03,215::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70538::DEBUG::2014-09-25 17:32:03,215::task::1191::Storage.TaskManager.Task::(prepare) Task=`0ae1de82-3179-42a0-bfdd-e6b06d6a0362`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70538::DEBUG::2014-09-25 17:32:03,215::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ae1de82-3179-42a0-bfdd-e6b06d6a0362`::moving from state preparing -> state finished >Thread-70538::DEBUG::2014-09-25 17:32:03,216::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70538::DEBUG::2014-09-25 17:32:03,216::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70538::DEBUG::2014-09-25 17:32:03,216::task::993::Storage.TaskManager.Task::(_decref) Task=`0ae1de82-3179-42a0-bfdd-e6b06d6a0362`::ref 0 aborting False >Thread-70538::DEBUG::2014-09-25 17:32:03,216::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70538::DEBUG::2014-09-25 17:32:03,217::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:03,228::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70539::DEBUG::2014-09-25 17:32:03,229::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:32:03,229::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70539::DEBUG::2014-09-25 17:32:03,231::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ac96e8f-8848-413b-ab16-b5166c09cfa1`::moving from state init -> state preparing >Thread-70539::INFO::2014-09-25 17:32:03,231::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70539::DEBUG::2014-09-25 17:32:03,232::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3fd03005-8d37-497d-9d7c-42f4fa55a190`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70539::DEBUG::2014-09-25 17:32:03,232::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70539::DEBUG::2014-09-25 17:32:03,232::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70539::DEBUG::2014-09-25 17:32:03,233::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3fd03005-8d37-497d-9d7c-42f4fa55a190`::Granted request >Thread-70539::DEBUG::2014-09-25 17:32:03,233::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`8ac96e8f-8848-413b-ab16-b5166c09cfa1`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70539::DEBUG::2014-09-25 17:32:03,233::task::993::Storage.TaskManager.Task::(_decref) Task=`8ac96e8f-8848-413b-ab16-b5166c09cfa1`::ref 1 aborting False >Thread-70539::INFO::2014-09-25 17:32:03,238::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70539::DEBUG::2014-09-25 17:32:03,238::task::1191::Storage.TaskManager.Task::(prepare) Task=`8ac96e8f-8848-413b-ab16-b5166c09cfa1`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70539::DEBUG::2014-09-25 17:32:03,238::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ac96e8f-8848-413b-ab16-b5166c09cfa1`::moving from state preparing -> state finished >Thread-70539::DEBUG::2014-09-25 17:32:03,239::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70539::DEBUG::2014-09-25 17:32:03,239::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70539::DEBUG::2014-09-25 17:32:03,239::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70539::DEBUG::2014-09-25 17:32:03,240::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70539::DEBUG::2014-09-25 17:32:03,240::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70539::DEBUG::2014-09-25 17:32:03,240::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70539::DEBUG::2014-09-25 17:32:03,240::task::993::Storage.TaskManager.Task::(_decref) Task=`8ac96e8f-8848-413b-ab16-b5166c09cfa1`::ref 0 aborting False >Thread-70539::DEBUG::2014-09-25 17:32:03,241::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70539::DEBUG::2014-09-25 17:32:03,241::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:32:04,560::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:04,561::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:04,561::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:04,561::__init__::232::IOProcess::(_processLogs) (108515) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:04,561::__init__::232::IOProcess::(_processLogs) (108515) Queuing response >Thread-80::DEBUG::2014-09-25 17:32:04,563::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:32:04,572::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000391735 s, 965 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:04,573::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:04,573::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:04,574::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:04,574::__init__::232::IOProcess::(_processLogs) (108516) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:04,576::__init__::232::IOProcess::(_processLogs) (108516) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:04,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:04,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0222124 s, 46.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:05,568::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:05,570::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70540::DEBUG::2014-09-25 17:32:05,572::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:32:05,781::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:05,781::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:05,781::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:05,782::__init__::232::IOProcess::(_processLogs) (108517) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:05,818::__init__::232::IOProcess::(_processLogs) (108517) Queuing response >Thread-79::DEBUG::2014-09-25 17:32:05,820::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:32:05,845::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00321673 s, 175 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:05,846::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:05,846::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:05,846::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:05,846::__init__::232::IOProcess::(_processLogs) (108518) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:05,851::__init__::232::IOProcess::(_processLogs) (108518) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:05,852::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:05,852::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:05,853::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:05,853::__init__::232::IOProcess::(_processLogs) (108519) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:05,854::__init__::232::IOProcess::(_processLogs) (108519) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:05,854::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:05,855::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:05,855::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:05,855::__init__::232::IOProcess::(_processLogs) (108520) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:05,856::__init__::232::IOProcess::(_processLogs) (108520) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:05,856::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:05,857::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:05,857::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:05,858::__init__::232::IOProcess::(_processLogs) (108521) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:05,858::__init__::232::IOProcess::(_processLogs) (108521) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:06,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:06,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132167 s, 77.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:08,585::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:08,587::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70541::DEBUG::2014-09-25 17:32:08,589::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:08,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:08,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162585 s, 63.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:10,804::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:10,804::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:10,804::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:10,804::__init__::232::IOProcess::(_processLogs) (108522) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:10,807::__init__::232::IOProcess::(_processLogs) (108522) Queuing response >Thread-81::DEBUG::2014-09-25 17:32:10,809::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:32:10,832::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00710481 s, 52.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:10,833::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:10,833::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:10,833::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:10,833::__init__::232::IOProcess::(_processLogs) (108523) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:10,841::__init__::232::IOProcess::(_processLogs) (108523) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:10,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:11,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021996 s, 46.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:11,606::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:11,607::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70542::DEBUG::2014-09-25 17:32:11,609::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:13,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:13,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167999 s, 61.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:13,267::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70543::DEBUG::2014-09-25 17:32:13,269::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:32:13,269::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70543::DEBUG::2014-09-25 17:32:13,271::task::595::Storage.TaskManager.Task::(_updateState) Task=`e666c176-d7fd-4816-9ede-4e316c5f2c60`::moving from state init -> state preparing >Thread-70543::INFO::2014-09-25 17:32:13,272::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70543::INFO::2014-09-25 17:32:13,272::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70543::DEBUG::2014-09-25 17:32:13,273::task::1191::Storage.TaskManager.Task::(prepare) Task=`e666c176-d7fd-4816-9ede-4e316c5f2c60`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70543::DEBUG::2014-09-25 17:32:13,273::task::595::Storage.TaskManager.Task::(_updateState) Task=`e666c176-d7fd-4816-9ede-4e316c5f2c60`::moving from state preparing -> state finished >Thread-70543::DEBUG::2014-09-25 17:32:13,273::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70543::DEBUG::2014-09-25 17:32:13,273::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70543::DEBUG::2014-09-25 17:32:13,274::task::993::Storage.TaskManager.Task::(_decref) Task=`e666c176-d7fd-4816-9ede-4e316c5f2c60`::ref 0 aborting False >Thread-70543::DEBUG::2014-09-25 17:32:13,274::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70543::DEBUG::2014-09-25 17:32:13,274::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:13,285::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70544::DEBUG::2014-09-25 17:32:13,286::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:32:13,286::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70544::DEBUG::2014-09-25 17:32:13,288::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a9febfe-f770-470a-baf7-4580c0cc6e72`::moving from state init -> state preparing >Thread-70544::INFO::2014-09-25 17:32:13,289::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70544::DEBUG::2014-09-25 17:32:13,289::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e4a250e9-977d-48fc-a0fe-8377799bf608`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70544::DEBUG::2014-09-25 17:32:13,290::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70544::DEBUG::2014-09-25 17:32:13,290::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70544::DEBUG::2014-09-25 17:32:13,290::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e4a250e9-977d-48fc-a0fe-8377799bf608`::Granted request >Thread-70544::DEBUG::2014-09-25 17:32:13,292::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0a9febfe-f770-470a-baf7-4580c0cc6e72`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70544::DEBUG::2014-09-25 17:32:13,292::task::993::Storage.TaskManager.Task::(_decref) Task=`0a9febfe-f770-470a-baf7-4580c0cc6e72`::ref 1 aborting False >Thread-70544::INFO::2014-09-25 17:32:13,297::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70544::DEBUG::2014-09-25 17:32:13,297::task::1191::Storage.TaskManager.Task::(prepare) Task=`0a9febfe-f770-470a-baf7-4580c0cc6e72`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70544::DEBUG::2014-09-25 17:32:13,297::task::595::Storage.TaskManager.Task::(_updateState) Task=`0a9febfe-f770-470a-baf7-4580c0cc6e72`::moving from state preparing -> state finished >Thread-70544::DEBUG::2014-09-25 17:32:13,297::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70544::DEBUG::2014-09-25 17:32:13,298::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70544::DEBUG::2014-09-25 17:32:13,298::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70544::DEBUG::2014-09-25 17:32:13,298::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70544::DEBUG::2014-09-25 17:32:13,299::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70544::DEBUG::2014-09-25 17:32:13,299::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70544::DEBUG::2014-09-25 17:32:13,299::task::993::Storage.TaskManager.Task::(_decref) Task=`0a9febfe-f770-470a-baf7-4580c0cc6e72`::ref 0 aborting False >Thread-70544::DEBUG::2014-09-25 17:32:13,300::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70544::DEBUG::2014-09-25 17:32:13,300::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:32:14,576::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:14,577::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:14,577::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:14,577::__init__::232::IOProcess::(_processLogs) (108524) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:14,578::__init__::232::IOProcess::(_processLogs) (108524) Queuing response >Thread-80::DEBUG::2014-09-25 17:32:14,579::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:32:14,588::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000449632 s, 841 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:14,589::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:14,590::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:14,590::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:14,592::__init__::232::IOProcess::(_processLogs) (108525) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:14,592::__init__::232::IOProcess::(_processLogs) (108525) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:14,626::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:14,627::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70545::DEBUG::2014-09-25 17:32:14,630::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a5ac483-747b-4079-9aac-ce15ca1335be`::moving from state init -> state preparing >Thread-70545::INFO::2014-09-25 17:32:14,630::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70545::INFO::2014-09-25 17:32:14,631::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00321673', 'lastCheck': '8.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449632', 'lastCheck': '0.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00710481', 'lastCheck': '3.8', 'valid': True}} >Thread-70545::DEBUG::2014-09-25 17:32:14,631::task::1191::Storage.TaskManager.Task::(prepare) Task=`9a5ac483-747b-4079-9aac-ce15ca1335be`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00321673', 'lastCheck': '8.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449632', 'lastCheck': '0.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00710481', 'lastCheck': '3.8', 'valid': True}} >Thread-70545::DEBUG::2014-09-25 17:32:14,631::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a5ac483-747b-4079-9aac-ce15ca1335be`::moving from state preparing -> state finished >Thread-70545::DEBUG::2014-09-25 17:32:14,631::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70545::DEBUG::2014-09-25 17:32:14,632::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70545::DEBUG::2014-09-25 17:32:14,632::task::993::Storage.TaskManager.Task::(_decref) Task=`9a5ac483-747b-4079-9aac-ce15ca1335be`::ref 0 aborting False >Thread-70545::DEBUG::2014-09-25 17:32:14,634::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:14,652::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:14,653::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70546::DEBUG::2014-09-25 17:32:14,655::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:15,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:15,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167751 s, 61.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:15,858::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:15,858::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:15,859::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:15,859::__init__::232::IOProcess::(_processLogs) (108526) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:15,860::__init__::232::IOProcess::(_processLogs) (108526) Queuing response >Thread-79::DEBUG::2014-09-25 17:32:15,861::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:32:15,870::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000358765 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:15,871::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:15,871::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:15,871::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:15,872::__init__::232::IOProcess::(_processLogs) (108527) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:15,872::__init__::232::IOProcess::(_processLogs) (108527) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:15,874::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:15,874::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:15,875::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:15,875::__init__::232::IOProcess::(_processLogs) (108528) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:15,876::__init__::232::IOProcess::(_processLogs) (108528) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:15,876::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:15,877::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:15,877::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:15,878::__init__::232::IOProcess::(_processLogs) (108529) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:15,878::__init__::232::IOProcess::(_processLogs) (108529) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:15,878::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:15,879::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:15,880::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:15,880::__init__::232::IOProcess::(_processLogs) (108530) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:15,880::__init__::232::IOProcess::(_processLogs) (108530) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:17,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:17,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189159 s, 54.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:17,699::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:17,700::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70547::DEBUG::2014-09-25 17:32:17,702::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:19,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:19,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0195478 s, 52.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:20,716::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:20,718::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70548::DEBUG::2014-09-25 17:32:20,720::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:32:20,844::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:20,844::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:20,844::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:20,845::__init__::232::IOProcess::(_processLogs) (108531) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:20,848::__init__::232::IOProcess::(_processLogs) (108531) Queuing response >Thread-81::DEBUG::2014-09-25 17:32:20,848::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:32:20,870::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00599983 s, 61.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:20,871::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:20,871::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:20,871::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:20,872::__init__::232::IOProcess::(_processLogs) (108532) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:20,882::__init__::232::IOProcess::(_processLogs) (108532) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:21,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:21,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164091 s, 62.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:32:23,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:23,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0222769 s, 46.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:23,328::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:23,331::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70549::DEBUG::2014-09-25 17:32:23,331::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70549::DEBUG::2014-09-25 17:32:23,338::task::595::Storage.TaskManager.Task::(_updateState) Task=`e262e4cc-faf8-47b8-abdb-ab55f2c394fe`::moving from state init -> state preparing >Thread-70549::INFO::2014-09-25 17:32:23,340::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70549::INFO::2014-09-25 17:32:23,343::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70549::DEBUG::2014-09-25 17:32:23,345::task::1191::Storage.TaskManager.Task::(prepare) Task=`e262e4cc-faf8-47b8-abdb-ab55f2c394fe`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70549::DEBUG::2014-09-25 17:32:23,346::task::595::Storage.TaskManager.Task::(_updateState) Task=`e262e4cc-faf8-47b8-abdb-ab55f2c394fe`::moving from state preparing -> state finished >Thread-70549::DEBUG::2014-09-25 17:32:23,348::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70549::DEBUG::2014-09-25 17:32:23,349::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70549::DEBUG::2014-09-25 17:32:23,351::task::993::Storage.TaskManager.Task::(_decref) Task=`e262e4cc-faf8-47b8-abdb-ab55f2c394fe`::ref 0 aborting False >Thread-70549::DEBUG::2014-09-25 17:32:23,353::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70549::DEBUG::2014-09-25 17:32:23,355::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:23,370::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70550::DEBUG::2014-09-25 17:32:23,371::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:32:23,371::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70550::DEBUG::2014-09-25 17:32:23,373::task::595::Storage.TaskManager.Task::(_updateState) Task=`1332bfc0-70ab-4c72-9886-26a0b88ddf6f`::moving from state init -> state preparing >Thread-70550::INFO::2014-09-25 17:32:23,373::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70550::DEBUG::2014-09-25 17:32:23,374::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1c8db1a8-c7b5-4aa7-9d0e-9158c3777c8d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70550::DEBUG::2014-09-25 17:32:23,374::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70550::DEBUG::2014-09-25 17:32:23,374::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70550::DEBUG::2014-09-25 17:32:23,375::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1c8db1a8-c7b5-4aa7-9d0e-9158c3777c8d`::Granted request >Thread-70550::DEBUG::2014-09-25 17:32:23,375::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1332bfc0-70ab-4c72-9886-26a0b88ddf6f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70550::DEBUG::2014-09-25 17:32:23,375::task::993::Storage.TaskManager.Task::(_decref) Task=`1332bfc0-70ab-4c72-9886-26a0b88ddf6f`::ref 1 aborting False >Thread-70550::INFO::2014-09-25 17:32:23,380::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70550::DEBUG::2014-09-25 17:32:23,380::task::1191::Storage.TaskManager.Task::(prepare) Task=`1332bfc0-70ab-4c72-9886-26a0b88ddf6f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70550::DEBUG::2014-09-25 17:32:23,380::task::595::Storage.TaskManager.Task::(_updateState) Task=`1332bfc0-70ab-4c72-9886-26a0b88ddf6f`::moving from state preparing -> state finished >Thread-70550::DEBUG::2014-09-25 17:32:23,380::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70550::DEBUG::2014-09-25 17:32:23,381::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70550::DEBUG::2014-09-25 17:32:23,381::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70550::DEBUG::2014-09-25 17:32:23,381::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70550::DEBUG::2014-09-25 17:32:23,382::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70550::DEBUG::2014-09-25 17:32:23,382::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70550::DEBUG::2014-09-25 17:32:23,382::task::993::Storage.TaskManager.Task::(_decref) Task=`1332bfc0-70ab-4c72-9886-26a0b88ddf6f`::ref 0 aborting False >Thread-70550::DEBUG::2014-09-25 17:32:23,383::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70550::DEBUG::2014-09-25 17:32:23,383::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:23,734::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:23,735::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70551::DEBUG::2014-09-25 17:32:23,737::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:32:24,593::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:24,593::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:24,593::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:24,594::__init__::232::IOProcess::(_processLogs) (108533) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:24,594::__init__::232::IOProcess::(_processLogs) (108533) Queuing response >Thread-80::DEBUG::2014-09-25 17:32:24,595::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:32:24,604::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000523595 s, 722 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:24,606::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:24,606::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:24,606::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:24,606::__init__::232::IOProcess::(_processLogs) (108534) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:24,607::__init__::232::IOProcess::(_processLogs) (108534) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:25,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:25,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213661 s, 47.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:25,881::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:25,881::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:25,881::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:25,881::__init__::232::IOProcess::(_processLogs) (108535) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:25,883::__init__::232::IOProcess::(_processLogs) (108535) Queuing response >Thread-79::DEBUG::2014-09-25 17:32:25,884::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:32:25,904::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00454334 s, 124 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:25,905::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:25,905::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:25,906::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:25,906::__init__::232::IOProcess::(_processLogs) (108536) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:25,913::__init__::232::IOProcess::(_processLogs) (108536) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:25,914::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:25,915::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:25,916::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:25,916::__init__::232::IOProcess::(_processLogs) (108537) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:25,916::__init__::232::IOProcess::(_processLogs) (108537) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:25,917::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:25,917::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:25,918::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:25,918::__init__::232::IOProcess::(_processLogs) (108538) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:25,918::__init__::232::IOProcess::(_processLogs) (108538) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:25,919::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:25,919::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:25,920::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:25,920::__init__::232::IOProcess::(_processLogs) (108539) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:25,921::__init__::232::IOProcess::(_processLogs) (108539) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:26,752::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:26,753::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70552::DEBUG::2014-09-25 17:32:26,755::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:27,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:27,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146149 s, 70.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:32:29,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:29,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134102 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:29,768::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:29,769::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70553::DEBUG::2014-09-25 17:32:29,772::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c62ad85-de76-4280-8ebd-26debcf3222c`::moving from state init -> state preparing >Thread-70553::INFO::2014-09-25 17:32:29,772::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70553::INFO::2014-09-25 17:32:29,773::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00454334', 'lastCheck': '3.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000523595', 'lastCheck': '5.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00599983', 'lastCheck': '8.9', 'valid': True}} >Thread-70553::DEBUG::2014-09-25 17:32:29,773::task::1191::Storage.TaskManager.Task::(prepare) Task=`6c62ad85-de76-4280-8ebd-26debcf3222c`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00454334', 'lastCheck': '3.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000523595', 'lastCheck': '5.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00599983', 'lastCheck': '8.9', 'valid': True}} >Thread-70553::DEBUG::2014-09-25 17:32:29,773::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c62ad85-de76-4280-8ebd-26debcf3222c`::moving from state preparing -> state finished >Thread-70553::DEBUG::2014-09-25 17:32:29,774::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70553::DEBUG::2014-09-25 17:32:29,774::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70553::DEBUG::2014-09-25 17:32:29,774::task::993::Storage.TaskManager.Task::(_decref) Task=`6c62ad85-de76-4280-8ebd-26debcf3222c`::ref 0 aborting False >Thread-70553::DEBUG::2014-09-25 17:32:29,776::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:29,796::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:29,800::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70554::DEBUG::2014-09-25 17:32:29,803::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 17:32:30,884::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:32:30,888::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:30,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:30,888::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:30,889::__init__::232::IOProcess::(_processLogs) (108540) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:32:30,889::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:30,889::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:30,890::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:30,890::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:30,890::__init__::232::IOProcess::(_processLogs) (108541) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:32:30,890::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:30,891::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:30,891::__init__::232::IOProcess::(_processLogs) (108542) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:32:30,891::__init__::232::IOProcess::(_processLogs) (108541) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:30,891::__init__::232::IOProcess::(_processLogs) (108540) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:30,893::__init__::232::IOProcess::(_processLogs) (108542) Queuing response >Thread-81::DEBUG::2014-09-25 17:32:30,898::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:32:30,899::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:30,899::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:30,899::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:30,900::__init__::232::IOProcess::(_processLogs) (108543) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:32:30,934::__init__::232::IOProcess::(_processLogs) (108543) Queuing response >Thread-81::DEBUG::2014-09-25 17:32:30,934::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:32:30,936::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:30,936::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:30,936::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:30,937::__init__::232::IOProcess::(_processLogs) (108544) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:32:30,945::__init__::232::IOProcess::(_processLogs) (108544) Queuing response >Thread-81::DEBUG::2014-09-25 17:32:30,945::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:32:30,946::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:30,947::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:30,947::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:30,947::__init__::232::IOProcess::(_processLogs) (108545) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:30,947::__init__::232::IOProcess::(_processLogs) (108545) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:30,948::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:30,948::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:30,949::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:30,949::__init__::232::IOProcess::(_processLogs) (108546) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:32:30,961::__init__::232::IOProcess::(_processLogs) (108546) Queuing response >Thread-81::DEBUG::2014-09-25 17:32:30,962::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 17:32:30,962::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:32:30,962::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:32:30,963::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:30,963::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:30,964::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:30,964::__init__::232::IOProcess::(_processLogs) (108547) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:30,972::__init__::232::IOProcess::(_processLogs) (108547) Queuing response >Thread-81::DEBUG::2014-09-25 17:32:30,973::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:32:30,998::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0086635 s, 42.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:30,999::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:30,999::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:31,000::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:31,000::__init__::232::IOProcess::(_processLogs) (108548) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:31,007::__init__::232::IOProcess::(_processLogs) (108548) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:31,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:31,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183798 s, 55.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:32,849::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:32,852::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70558::DEBUG::2014-09-25 17:32:32,857::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:33,411::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:33,413::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70559::DEBUG::2014-09-25 17:32:33,413::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70559::DEBUG::2014-09-25 17:32:33,415::task::595::Storage.TaskManager.Task::(_updateState) Task=`30d126d8-0e94-4a1e-bf4a-0e2a03447615`::moving from state init -> state preparing >Thread-70559::INFO::2014-09-25 17:32:33,415::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70559::INFO::2014-09-25 17:32:33,416::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70559::DEBUG::2014-09-25 17:32:33,416::task::1191::Storage.TaskManager.Task::(prepare) Task=`30d126d8-0e94-4a1e-bf4a-0e2a03447615`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70559::DEBUG::2014-09-25 17:32:33,416::task::595::Storage.TaskManager.Task::(_updateState) Task=`30d126d8-0e94-4a1e-bf4a-0e2a03447615`::moving from state preparing -> state finished >Thread-70559::DEBUG::2014-09-25 17:32:33,416::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70559::DEBUG::2014-09-25 17:32:33,416::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70559::DEBUG::2014-09-25 17:32:33,417::task::993::Storage.TaskManager.Task::(_decref) Task=`30d126d8-0e94-4a1e-bf4a-0e2a03447615`::ref 0 aborting False >Thread-70559::DEBUG::2014-09-25 17:32:33,417::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70559::DEBUG::2014-09-25 17:32:33,417::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:33,431::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:33,433::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70560::DEBUG::2014-09-25 17:32:33,433::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70560::DEBUG::2014-09-25 17:32:33,434::task::595::Storage.TaskManager.Task::(_updateState) Task=`3cfeb6e4-2d59-4111-9e4d-7175d1c3b093`::moving from state init -> state preparing >Thread-70560::INFO::2014-09-25 17:32:33,435::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70560::DEBUG::2014-09-25 17:32:33,435::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2a2e2e40-0739-4333-854a-2bf44697cec6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70560::DEBUG::2014-09-25 17:32:33,436::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70560::DEBUG::2014-09-25 17:32:33,436::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70560::DEBUG::2014-09-25 17:32:33,436::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2a2e2e40-0739-4333-854a-2bf44697cec6`::Granted request >Thread-70560::DEBUG::2014-09-25 17:32:33,437::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3cfeb6e4-2d59-4111-9e4d-7175d1c3b093`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70560::DEBUG::2014-09-25 17:32:33,437::task::993::Storage.TaskManager.Task::(_decref) Task=`3cfeb6e4-2d59-4111-9e4d-7175d1c3b093`::ref 1 aborting False >Thread-70560::INFO::2014-09-25 17:32:33,441::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70560::DEBUG::2014-09-25 17:32:33,442::task::1191::Storage.TaskManager.Task::(prepare) Task=`3cfeb6e4-2d59-4111-9e4d-7175d1c3b093`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70560::DEBUG::2014-09-25 17:32:33,442::task::595::Storage.TaskManager.Task::(_updateState) Task=`3cfeb6e4-2d59-4111-9e4d-7175d1c3b093`::moving from state preparing -> state finished >Thread-70560::DEBUG::2014-09-25 17:32:33,442::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70560::DEBUG::2014-09-25 17:32:33,442::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70560::DEBUG::2014-09-25 17:32:33,443::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70560::DEBUG::2014-09-25 17:32:33,443::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70560::DEBUG::2014-09-25 17:32:33,443::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70560::DEBUG::2014-09-25 17:32:33,444::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70560::DEBUG::2014-09-25 17:32:33,444::task::993::Storage.TaskManager.Task::(_decref) Task=`3cfeb6e4-2d59-4111-9e4d-7175d1c3b093`::ref 0 aborting False >Thread-70560::DEBUG::2014-09-25 17:32:33,444::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70560::DEBUG::2014-09-25 17:32:33,445::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:33,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:33,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131418 s, 77.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:34,609::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:34,610::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:34,610::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:34,610::__init__::232::IOProcess::(_processLogs) (108549) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:34,610::__init__::232::IOProcess::(_processLogs) (108549) Queuing response >Thread-80::DEBUG::2014-09-25 17:32:34,611::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:32:34,620::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000516457 s, 732 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:34,621::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:34,621::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:34,622::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:34,622::__init__::232::IOProcess::(_processLogs) (108550) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:34,622::__init__::232::IOProcess::(_processLogs) (108550) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:35,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:35,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141223 s, 72.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:35,869::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:35,871::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70561::DEBUG::2014-09-25 17:32:35,873::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:32:35,921::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:35,921::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:35,921::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:35,922::__init__::232::IOProcess::(_processLogs) (108551) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:35,941::__init__::232::IOProcess::(_processLogs) (108551) Queuing response >Thread-79::DEBUG::2014-09-25 17:32:35,942::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:32:35,966::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00644406 s, 87.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:35,968::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:35,968::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:35,968::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:35,968::__init__::232::IOProcess::(_processLogs) (108552) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:35,971::__init__::232::IOProcess::(_processLogs) (108552) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:35,973::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:35,973::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:35,973::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:35,974::__init__::232::IOProcess::(_processLogs) (108553) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:35,974::__init__::232::IOProcess::(_processLogs) (108553) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:35,975::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:35,975::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:35,975::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:35,976::__init__::232::IOProcess::(_processLogs) (108554) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:35,976::__init__::232::IOProcess::(_processLogs) (108554) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:35,977::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:35,977::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:35,978::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:35,978::__init__::232::IOProcess::(_processLogs) (108555) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:35,979::__init__::232::IOProcess::(_processLogs) (108555) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:37,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:37,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155288 s, 65.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:38,883::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:38,885::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70562::DEBUG::2014-09-25 17:32:38,887::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:39,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:39,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132459 s, 77.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:41,009::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:41,010::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:41,010::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:32:41,012::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:32:41,012::__init__::232::IOProcess::(_processLogs) (108556) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:41,020::__init__::232::IOProcess::(_processLogs) (108556) Queuing response >Thread-81::DEBUG::2014-09-25 17:32:41,021::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000453229 s, 819 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:41,022::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:41,022::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:41,023::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:41,023::__init__::232::IOProcess::(_processLogs) (108557) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:41,023::__init__::232::IOProcess::(_processLogs) (108557) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:41,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:41,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0197339 s, 51.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:41,899::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:41,900::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70563::DEBUG::2014-09-25 17:32:41,902::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:43,474::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70564::DEBUG::2014-09-25 17:32:43,475::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:32:43,475::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70564::DEBUG::2014-09-25 17:32:43,477::task::595::Storage.TaskManager.Task::(_updateState) Task=`7b9a56f7-e430-4e15-a0a8-cf3282db271a`::moving from state init -> state preparing >Thread-70564::INFO::2014-09-25 17:32:43,478::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70564::INFO::2014-09-25 17:32:43,478::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70564::DEBUG::2014-09-25 17:32:43,478::task::1191::Storage.TaskManager.Task::(prepare) Task=`7b9a56f7-e430-4e15-a0a8-cf3282db271a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70564::DEBUG::2014-09-25 17:32:43,479::task::595::Storage.TaskManager.Task::(_updateState) Task=`7b9a56f7-e430-4e15-a0a8-cf3282db271a`::moving from state preparing -> state finished >Thread-70564::DEBUG::2014-09-25 17:32:43,479::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70564::DEBUG::2014-09-25 17:32:43,479::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70564::DEBUG::2014-09-25 17:32:43,479::task::993::Storage.TaskManager.Task::(_decref) Task=`7b9a56f7-e430-4e15-a0a8-cf3282db271a`::ref 0 aborting False >Thread-70564::DEBUG::2014-09-25 17:32:43,480::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70564::DEBUG::2014-09-25 17:32:43,480::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:43,490::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70565::DEBUG::2014-09-25 17:32:43,492::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:32:43,492::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70565::DEBUG::2014-09-25 17:32:43,494::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e5dee70-f5d0-4033-8a3f-5178bc33dfc6`::moving from state init -> state preparing >Thread-70565::INFO::2014-09-25 17:32:43,494::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70565::DEBUG::2014-09-25 17:32:43,495::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`522136bf-3f70-4136-bf1d-28e061f06b95`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70565::DEBUG::2014-09-25 17:32:43,495::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70565::DEBUG::2014-09-25 17:32:43,495::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70565::DEBUG::2014-09-25 17:32:43,495::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`522136bf-3f70-4136-bf1d-28e061f06b95`::Granted request >Thread-70565::DEBUG::2014-09-25 17:32:43,496::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5e5dee70-f5d0-4033-8a3f-5178bc33dfc6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70565::DEBUG::2014-09-25 17:32:43,496::task::993::Storage.TaskManager.Task::(_decref) Task=`5e5dee70-f5d0-4033-8a3f-5178bc33dfc6`::ref 1 aborting False >Thread-70565::INFO::2014-09-25 17:32:43,501::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70565::DEBUG::2014-09-25 17:32:43,501::task::1191::Storage.TaskManager.Task::(prepare) Task=`5e5dee70-f5d0-4033-8a3f-5178bc33dfc6`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70565::DEBUG::2014-09-25 17:32:43,501::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e5dee70-f5d0-4033-8a3f-5178bc33dfc6`::moving from state preparing -> state finished >Thread-70565::DEBUG::2014-09-25 17:32:43,501::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70565::DEBUG::2014-09-25 17:32:43,502::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70565::DEBUG::2014-09-25 17:32:43,502::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70565::DEBUG::2014-09-25 17:32:43,502::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70565::DEBUG::2014-09-25 17:32:43,503::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70565::DEBUG::2014-09-25 17:32:43,503::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70565::DEBUG::2014-09-25 17:32:43,503::task::993::Storage.TaskManager.Task::(_decref) Task=`5e5dee70-f5d0-4033-8a3f-5178bc33dfc6`::ref 0 aborting False >Thread-70565::DEBUG::2014-09-25 17:32:43,504::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70565::DEBUG::2014-09-25 17:32:43,504::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:43,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:43,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133809 s, 76.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:44,625::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:44,625::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:44,625::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:44,626::__init__::232::IOProcess::(_processLogs) (108558) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:44,626::__init__::232::IOProcess::(_processLogs) (108558) Queuing response >Thread-80::DEBUG::2014-09-25 17:32:44,628::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:32:44,637::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000430149 s, 879 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:44,638::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:44,638::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:44,639::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:44,639::__init__::232::IOProcess::(_processLogs) (108559) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:44,640::__init__::232::IOProcess::(_processLogs) (108559) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:44,913::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:44,914::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70566::DEBUG::2014-09-25 17:32:44,917::task::595::Storage.TaskManager.Task::(_updateState) Task=`46c4c5d7-201a-4dd7-8b63-27f1dcaee6f1`::moving from state init -> state preparing >Thread-70566::INFO::2014-09-25 17:32:44,917::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70566::INFO::2014-09-25 17:32:44,918::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00644406', 'lastCheck': '8.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000430149', 'lastCheck': '0.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000453229', 'lastCheck': '3.9', 'valid': True}} >Thread-70566::DEBUG::2014-09-25 17:32:44,918::task::1191::Storage.TaskManager.Task::(prepare) Task=`46c4c5d7-201a-4dd7-8b63-27f1dcaee6f1`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00644406', 'lastCheck': '8.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000430149', 'lastCheck': '0.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000453229', 'lastCheck': '3.9', 'valid': True}} >Thread-70566::DEBUG::2014-09-25 17:32:44,918::task::595::Storage.TaskManager.Task::(_updateState) Task=`46c4c5d7-201a-4dd7-8b63-27f1dcaee6f1`::moving from state preparing -> state finished >Thread-70566::DEBUG::2014-09-25 17:32:44,919::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70566::DEBUG::2014-09-25 17:32:44,919::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70566::DEBUG::2014-09-25 17:32:44,919::task::993::Storage.TaskManager.Task::(_decref) Task=`46c4c5d7-201a-4dd7-8b63-27f1dcaee6f1`::ref 0 aborting False >Thread-70566::DEBUG::2014-09-25 17:32:44,922::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:44,937::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:44,938::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70567::DEBUG::2014-09-25 17:32:44,940::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:45,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:45,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152729 s, 67.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:45,979::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:45,979::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:45,980::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:45,980::__init__::232::IOProcess::(_processLogs) (108560) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:45,981::__init__::232::IOProcess::(_processLogs) (108560) Queuing response >Thread-79::DEBUG::2014-09-25 17:32:45,982::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:32:45,997::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00517262 s, 109 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:45,999::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:45,999::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:45,999::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:46,000::__init__::232::IOProcess::(_processLogs) (108561) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:46,003::__init__::232::IOProcess::(_processLogs) (108561) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:46,004::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:46,004::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:46,005::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:46,005::__init__::232::IOProcess::(_processLogs) (108562) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:46,008::__init__::232::IOProcess::(_processLogs) (108562) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:46,009::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:46,009::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:46,009::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:46,010::__init__::232::IOProcess::(_processLogs) (108563) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:46,016::__init__::232::IOProcess::(_processLogs) (108563) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:46,018::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:46,019::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:46,020::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:46,022::__init__::232::IOProcess::(_processLogs) (108564) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:46,024::__init__::232::IOProcess::(_processLogs) (108564) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:47,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:47,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131912 s, 77.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:47,983::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:47,985::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70568::DEBUG::2014-09-25 17:32:47,987::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:49,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:49,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177537 s, 57.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:50,999::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:51,000::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70569::DEBUG::2014-09-25 17:32:51,002::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:32:51,026::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:51,026::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:51,026::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:51,026::__init__::232::IOProcess::(_processLogs) (108565) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:51,031::__init__::232::IOProcess::(_processLogs) (108565) Queuing response >Thread-81::DEBUG::2014-09-25 17:32:51,032::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:32:51,049::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00284668 s, 130 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:51,051::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:51,051::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:51,051::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:51,051::__init__::232::IOProcess::(_processLogs) (108566) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:51,057::__init__::232::IOProcess::(_processLogs) (108566) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:51,976::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:52,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.219928 s, 4.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:53,527::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70570::DEBUG::2014-09-25 17:32:53,528::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:32:53,528::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70570::DEBUG::2014-09-25 17:32:53,530::task::595::Storage.TaskManager.Task::(_updateState) Task=`270854b9-fd83-4402-aa59-95102d72c15d`::moving from state init -> state preparing >Thread-70570::INFO::2014-09-25 17:32:53,531::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70570::INFO::2014-09-25 17:32:53,531::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70570::DEBUG::2014-09-25 17:32:53,532::task::1191::Storage.TaskManager.Task::(prepare) Task=`270854b9-fd83-4402-aa59-95102d72c15d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70570::DEBUG::2014-09-25 17:32:53,532::task::595::Storage.TaskManager.Task::(_updateState) Task=`270854b9-fd83-4402-aa59-95102d72c15d`::moving from state preparing -> state finished >Thread-70570::DEBUG::2014-09-25 17:32:53,532::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70570::DEBUG::2014-09-25 17:32:53,532::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70570::DEBUG::2014-09-25 17:32:53,532::task::993::Storage.TaskManager.Task::(_decref) Task=`270854b9-fd83-4402-aa59-95102d72c15d`::ref 0 aborting False >Thread-70570::DEBUG::2014-09-25 17:32:53,533::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70570::DEBUG::2014-09-25 17:32:53,533::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:53,544::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:53,545::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70571::DEBUG::2014-09-25 17:32:53,545::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70571::DEBUG::2014-09-25 17:32:53,547::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ff48967-2c25-4ae3-b2bd-1772e6aea361`::moving from state init -> state preparing >Thread-70571::INFO::2014-09-25 17:32:53,547::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70571::DEBUG::2014-09-25 17:32:53,548::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`675ce9e7-5b4f-4de7-bd61-6962580d535b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70571::DEBUG::2014-09-25 17:32:53,548::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70571::DEBUG::2014-09-25 17:32:53,549::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70571::DEBUG::2014-09-25 17:32:53,549::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`675ce9e7-5b4f-4de7-bd61-6962580d535b`::Granted request >Thread-70571::DEBUG::2014-09-25 17:32:53,549::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4ff48967-2c25-4ae3-b2bd-1772e6aea361`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70571::DEBUG::2014-09-25 17:32:53,550::task::993::Storage.TaskManager.Task::(_decref) Task=`4ff48967-2c25-4ae3-b2bd-1772e6aea361`::ref 1 aborting False >Thread-70571::INFO::2014-09-25 17:32:53,554::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70571::DEBUG::2014-09-25 17:32:53,554::task::1191::Storage.TaskManager.Task::(prepare) Task=`4ff48967-2c25-4ae3-b2bd-1772e6aea361`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70571::DEBUG::2014-09-25 17:32:53,555::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ff48967-2c25-4ae3-b2bd-1772e6aea361`::moving from state preparing -> state finished >Thread-70571::DEBUG::2014-09-25 17:32:53,555::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70571::DEBUG::2014-09-25 17:32:53,555::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70571::DEBUG::2014-09-25 17:32:53,555::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70571::DEBUG::2014-09-25 17:32:53,556::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70571::DEBUG::2014-09-25 17:32:53,556::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70571::DEBUG::2014-09-25 17:32:53,556::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70571::DEBUG::2014-09-25 17:32:53,557::task::993::Storage.TaskManager.Task::(_decref) Task=`4ff48967-2c25-4ae3-b2bd-1772e6aea361`::ref 0 aborting False >Thread-70571::DEBUG::2014-09-25 17:32:53,557::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70571::DEBUG::2014-09-25 17:32:53,557::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:54,015::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:54,016::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70572::DEBUG::2014-09-25 17:32:54,018::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:54,226::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:54,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137319 s, 74.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:54,642::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:54,642::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:54,642::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:54,642::__init__::232::IOProcess::(_processLogs) (108567) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:54,643::__init__::232::IOProcess::(_processLogs) (108567) Queuing response >Thread-80::DEBUG::2014-09-25 17:32:54,644::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:32:54,653::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000514455 s, 735 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:54,654::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:54,654::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:54,654::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:54,655::__init__::232::IOProcess::(_processLogs) (108568) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:54,655::__init__::232::IOProcess::(_processLogs) (108568) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:56,026::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:56,026::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:56,026::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:56,026::__init__::232::IOProcess::(_processLogs) (108569) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:56,031::__init__::232::IOProcess::(_processLogs) (108569) Queuing response >Thread-79::DEBUG::2014-09-25 17:32:56,032::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:32:56,051::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00394694 s, 142 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:32:56,053::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:56,053::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:56,053::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:56,053::__init__::232::IOProcess::(_processLogs) (108570) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:32:56,057::__init__::232::IOProcess::(_processLogs) (108570) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:56,058::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:56,058::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:56,059::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:56,059::__init__::232::IOProcess::(_processLogs) (108571) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:56,059::__init__::232::IOProcess::(_processLogs) (108571) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:56,060::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:56,060::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:56,061::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:56,061::__init__::232::IOProcess::(_processLogs) (108572) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:56,061::__init__::232::IOProcess::(_processLogs) (108572) Queuing response >Thread-29::DEBUG::2014-09-25 17:32:56,062::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:32:56,063::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:32:56,063::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:32:56,064::__init__::232::IOProcess::(_processLogs) (108573) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:32:56,064::__init__::232::IOProcess::(_processLogs) (108573) Queuing response >Dummy-148::DEBUG::2014-09-25 17:32:56,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:56,508::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.223373 s, 4.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:32:57,031::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:32:57,032::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70573::DEBUG::2014-09-25 17:32:57,034::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:32:58,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:32:58,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200439 s, 51.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:00,050::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:00,051::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70574::DEBUG::2014-09-25 17:33:00,054::task::595::Storage.TaskManager.Task::(_updateState) Task=`d0824ce6-bf6d-4655-a30e-f5f82ebce712`::moving from state init -> state preparing >Thread-70574::INFO::2014-09-25 17:33:00,055::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70574::INFO::2014-09-25 17:33:00,055::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00394694', 'lastCheck': '4.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000514455', 'lastCheck': '5.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00284668', 'lastCheck': '9.0', 'valid': True}} >Thread-70574::DEBUG::2014-09-25 17:33:00,055::task::1191::Storage.TaskManager.Task::(prepare) Task=`d0824ce6-bf6d-4655-a30e-f5f82ebce712`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00394694', 'lastCheck': '4.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000514455', 'lastCheck': '5.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00284668', 'lastCheck': '9.0', 'valid': True}} >Thread-70574::DEBUG::2014-09-25 17:33:00,056::task::595::Storage.TaskManager.Task::(_updateState) Task=`d0824ce6-bf6d-4655-a30e-f5f82ebce712`::moving from state preparing -> state finished >Thread-70574::DEBUG::2014-09-25 17:33:00,056::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70574::DEBUG::2014-09-25 17:33:00,056::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70574::DEBUG::2014-09-25 17:33:00,056::task::993::Storage.TaskManager.Task::(_decref) Task=`d0824ce6-bf6d-4655-a30e-f5f82ebce712`::ref 0 aborting False >Thread-70574::DEBUG::2014-09-25 17:33:00,059::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:00,078::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:00,080::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70575::DEBUG::2014-09-25 17:33:00,081::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:00,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:00,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160316 s, 63.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:01,058::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:01,059::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:01,059::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:01,059::__init__::232::IOProcess::(_processLogs) (108574) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:01,060::__init__::232::IOProcess::(_processLogs) (108574) Queuing response >Thread-81::DEBUG::2014-09-25 17:33:01,060::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:33:01,069::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000434412 s, 854 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:01,071::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:01,071::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:01,071::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:01,071::__init__::232::IOProcess::(_processLogs) (108575) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:01,072::__init__::232::IOProcess::(_processLogs) (108575) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:02,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:02,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134327 s, 76.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:03,139::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:03,140::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70576::DEBUG::2014-09-25 17:33:03,142::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:03,582::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70577::DEBUG::2014-09-25 17:33:03,583::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:33:03,583::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70577::DEBUG::2014-09-25 17:33:03,585::task::595::Storage.TaskManager.Task::(_updateState) Task=`e6349169-bcca-467d-9244-fbf6ade21ae5`::moving from state init -> state preparing >Thread-70577::INFO::2014-09-25 17:33:03,586::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70577::INFO::2014-09-25 17:33:03,586::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70577::DEBUG::2014-09-25 17:33:03,587::task::1191::Storage.TaskManager.Task::(prepare) Task=`e6349169-bcca-467d-9244-fbf6ade21ae5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70577::DEBUG::2014-09-25 17:33:03,587::task::595::Storage.TaskManager.Task::(_updateState) Task=`e6349169-bcca-467d-9244-fbf6ade21ae5`::moving from state preparing -> state finished >Thread-70577::DEBUG::2014-09-25 17:33:03,587::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70577::DEBUG::2014-09-25 17:33:03,587::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70577::DEBUG::2014-09-25 17:33:03,587::task::993::Storage.TaskManager.Task::(_decref) Task=`e6349169-bcca-467d-9244-fbf6ade21ae5`::ref 0 aborting False >Thread-70577::DEBUG::2014-09-25 17:33:03,588::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70577::DEBUG::2014-09-25 17:33:03,588::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:03,603::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:03,604::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70578::DEBUG::2014-09-25 17:33:03,604::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70578::DEBUG::2014-09-25 17:33:03,606::task::595::Storage.TaskManager.Task::(_updateState) Task=`07cfe366-fda8-45bf-b3f2-0035a3602362`::moving from state init -> state preparing >Thread-70578::INFO::2014-09-25 17:33:03,606::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70578::DEBUG::2014-09-25 17:33:03,607::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3a729e86-f1f0-4ffe-964a-598edadb61e5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70578::DEBUG::2014-09-25 17:33:03,607::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70578::DEBUG::2014-09-25 17:33:03,607::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70578::DEBUG::2014-09-25 17:33:03,608::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3a729e86-f1f0-4ffe-964a-598edadb61e5`::Granted request >Thread-70578::DEBUG::2014-09-25 17:33:03,608::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`07cfe366-fda8-45bf-b3f2-0035a3602362`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70578::DEBUG::2014-09-25 17:33:03,609::task::993::Storage.TaskManager.Task::(_decref) Task=`07cfe366-fda8-45bf-b3f2-0035a3602362`::ref 1 aborting False >Thread-70578::INFO::2014-09-25 17:33:03,613::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70578::DEBUG::2014-09-25 17:33:03,613::task::1191::Storage.TaskManager.Task::(prepare) Task=`07cfe366-fda8-45bf-b3f2-0035a3602362`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70578::DEBUG::2014-09-25 17:33:03,613::task::595::Storage.TaskManager.Task::(_updateState) Task=`07cfe366-fda8-45bf-b3f2-0035a3602362`::moving from state preparing -> state finished >Thread-70578::DEBUG::2014-09-25 17:33:03,614::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70578::DEBUG::2014-09-25 17:33:03,614::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70578::DEBUG::2014-09-25 17:33:03,614::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70578::DEBUG::2014-09-25 17:33:03,615::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70578::DEBUG::2014-09-25 17:33:03,615::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70578::DEBUG::2014-09-25 17:33:03,615::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70578::DEBUG::2014-09-25 17:33:03,615::task::993::Storage.TaskManager.Task::(_decref) Task=`07cfe366-fda8-45bf-b3f2-0035a3602362`::ref 0 aborting False >Thread-70578::DEBUG::2014-09-25 17:33:03,616::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70578::DEBUG::2014-09-25 17:33:03,616::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:33:04,657::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:04,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:04,658::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:04,658::__init__::232::IOProcess::(_processLogs) (108576) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:04,659::__init__::232::IOProcess::(_processLogs) (108576) Queuing response >Thread-80::DEBUG::2014-09-25 17:33:04,660::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:33:04,669::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000405339 s, 933 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:04,670::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:04,671::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:04,671::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:04,672::__init__::232::IOProcess::(_processLogs) (108577) Got request for method 'statvfs' >Dummy-148::DEBUG::2014-09-25 17:33:04,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:33:04,674::__init__::232::IOProcess::(_processLogs) (108577) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:04,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203956 s, 50.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:06,065::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:06,065::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:06,065::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:06,066::__init__::232::IOProcess::(_processLogs) (108578) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:06,068::__init__::232::IOProcess::(_processLogs) (108578) Queuing response >Thread-79::DEBUG::2014-09-25 17:33:06,070::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:33:06,086::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00140709 s, 399 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:06,087::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:06,087::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:06,088::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:06,088::__init__::232::IOProcess::(_processLogs) (108579) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:06,095::__init__::232::IOProcess::(_processLogs) (108579) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:06,096::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:06,096::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:06,097::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:06,098::__init__::232::IOProcess::(_processLogs) (108580) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:06,098::__init__::232::IOProcess::(_processLogs) (108580) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:06,098::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:06,099::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:06,099::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:06,100::__init__::232::IOProcess::(_processLogs) (108581) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:06,100::__init__::232::IOProcess::(_processLogs) (108581) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:06,101::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:06,101::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:06,101::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:06,102::__init__::232::IOProcess::(_processLogs) (108582) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:06,103::__init__::232::IOProcess::(_processLogs) (108582) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:06,156::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:06,157::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70579::DEBUG::2014-09-25 17:33:06,159::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:06,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:06,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177318 s, 57.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:33:08,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:08,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134609 s, 76.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:09,172::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:09,173::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70580::DEBUG::2014-09-25 17:33:09,176::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:10,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:10,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181894 s, 56.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:11,074::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:11,075::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:11,075::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:11,075::__init__::232::IOProcess::(_processLogs) (108583) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:11,083::__init__::232::IOProcess::(_processLogs) (108583) Queuing response >Thread-81::DEBUG::2014-09-25 17:33:11,084::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:33:11,105::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00837082 s, 44.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:11,106::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:11,106::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:11,106::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:11,107::__init__::232::IOProcess::(_processLogs) (108584) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:11,109::__init__::232::IOProcess::(_processLogs) (108584) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:12,192::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:12,193::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70581::DEBUG::2014-09-25 17:33:12,195::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:12,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:12,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150562 s, 68.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:13,646::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70582::DEBUG::2014-09-25 17:33:13,647::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:33:13,647::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70582::DEBUG::2014-09-25 17:33:13,649::task::595::Storage.TaskManager.Task::(_updateState) Task=`4c5cd1e4-8927-43da-9256-e3033d01b28d`::moving from state init -> state preparing >Thread-70582::INFO::2014-09-25 17:33:13,650::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70582::INFO::2014-09-25 17:33:13,650::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70582::DEBUG::2014-09-25 17:33:13,651::task::1191::Storage.TaskManager.Task::(prepare) Task=`4c5cd1e4-8927-43da-9256-e3033d01b28d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70582::DEBUG::2014-09-25 17:33:13,651::task::595::Storage.TaskManager.Task::(_updateState) Task=`4c5cd1e4-8927-43da-9256-e3033d01b28d`::moving from state preparing -> state finished >Thread-70582::DEBUG::2014-09-25 17:33:13,651::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70582::DEBUG::2014-09-25 17:33:13,651::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70582::DEBUG::2014-09-25 17:33:13,651::task::993::Storage.TaskManager.Task::(_decref) Task=`4c5cd1e4-8927-43da-9256-e3033d01b28d`::ref 0 aborting False >Thread-70582::DEBUG::2014-09-25 17:33:13,652::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70582::DEBUG::2014-09-25 17:33:13,652::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:13,662::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70583::DEBUG::2014-09-25 17:33:13,663::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:33:13,663::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70583::DEBUG::2014-09-25 17:33:13,665::task::595::Storage.TaskManager.Task::(_updateState) Task=`62fc0533-ff58-4392-b250-ec2dc7f2ca17`::moving from state init -> state preparing >Thread-70583::INFO::2014-09-25 17:33:13,665::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70583::DEBUG::2014-09-25 17:33:13,666::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a17ef186-6d39-4a66-8c01-fcb9643711fb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70583::DEBUG::2014-09-25 17:33:13,666::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70583::DEBUG::2014-09-25 17:33:13,667::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70583::DEBUG::2014-09-25 17:33:13,667::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a17ef186-6d39-4a66-8c01-fcb9643711fb`::Granted request >Thread-70583::DEBUG::2014-09-25 17:33:13,668::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`62fc0533-ff58-4392-b250-ec2dc7f2ca17`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70583::DEBUG::2014-09-25 17:33:13,668::task::993::Storage.TaskManager.Task::(_decref) Task=`62fc0533-ff58-4392-b250-ec2dc7f2ca17`::ref 1 aborting False >Thread-70583::INFO::2014-09-25 17:33:13,672::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70583::DEBUG::2014-09-25 17:33:13,673::task::1191::Storage.TaskManager.Task::(prepare) Task=`62fc0533-ff58-4392-b250-ec2dc7f2ca17`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70583::DEBUG::2014-09-25 17:33:13,673::task::595::Storage.TaskManager.Task::(_updateState) Task=`62fc0533-ff58-4392-b250-ec2dc7f2ca17`::moving from state preparing -> state finished >Thread-70583::DEBUG::2014-09-25 17:33:13,673::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70583::DEBUG::2014-09-25 17:33:13,673::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70583::DEBUG::2014-09-25 17:33:13,674::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70583::DEBUG::2014-09-25 17:33:13,674::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70583::DEBUG::2014-09-25 17:33:13,674::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70583::DEBUG::2014-09-25 17:33:13,674::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70583::DEBUG::2014-09-25 17:33:13,675::task::993::Storage.TaskManager.Task::(_decref) Task=`62fc0533-ff58-4392-b250-ec2dc7f2ca17`::ref 0 aborting False >Thread-70583::DEBUG::2014-09-25 17:33:13,675::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70583::DEBUG::2014-09-25 17:33:13,676::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:33:14,684::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:14,684::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:14,684::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:14,685::__init__::232::IOProcess::(_processLogs) (108585) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:14,685::__init__::232::IOProcess::(_processLogs) (108585) Queuing response >Thread-80::DEBUG::2014-09-25 17:33:14,686::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:33:14,695::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000566507 s, 667 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:14,697::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:14,697::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:14,697::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:14,697::__init__::232::IOProcess::(_processLogs) (108586) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:14,698::__init__::232::IOProcess::(_processLogs) (108586) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:14,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:14,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018371 s, 55.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:15,209::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:15,211::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70584::DEBUG::2014-09-25 17:33:15,214::task::595::Storage.TaskManager.Task::(_updateState) Task=`56757e7b-1ade-4fc1-af9f-bd6de1de9754`::moving from state init -> state preparing >Thread-70584::INFO::2014-09-25 17:33:15,214::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70584::INFO::2014-09-25 17:33:15,215::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00140709', 'lastCheck': '9.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000566507', 'lastCheck': '0.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00837082', 'lastCheck': '4.1', 'valid': True}} >Thread-70584::DEBUG::2014-09-25 17:33:15,215::task::1191::Storage.TaskManager.Task::(prepare) Task=`56757e7b-1ade-4fc1-af9f-bd6de1de9754`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00140709', 'lastCheck': '9.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000566507', 'lastCheck': '0.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00837082', 'lastCheck': '4.1', 'valid': True}} >Thread-70584::DEBUG::2014-09-25 17:33:15,215::task::595::Storage.TaskManager.Task::(_updateState) Task=`56757e7b-1ade-4fc1-af9f-bd6de1de9754`::moving from state preparing -> state finished >Thread-70584::DEBUG::2014-09-25 17:33:15,215::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70584::DEBUG::2014-09-25 17:33:15,216::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70584::DEBUG::2014-09-25 17:33:15,216::task::993::Storage.TaskManager.Task::(_decref) Task=`56757e7b-1ade-4fc1-af9f-bd6de1de9754`::ref 0 aborting False >Thread-70584::DEBUG::2014-09-25 17:33:15,219::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:15,240::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:15,241::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70585::DEBUG::2014-09-25 17:33:15,243::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:33:16,104::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:16,106::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:16,107::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:16,108::__init__::232::IOProcess::(_processLogs) (108587) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:16,118::__init__::232::IOProcess::(_processLogs) (108587) Queuing response >Thread-79::DEBUG::2014-09-25 17:33:16,120::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:33:16,151::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00614228 s, 91.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:16,154::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:16,155::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:16,156::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:16,157::__init__::232::IOProcess::(_processLogs) (108588) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:16,159::__init__::232::IOProcess::(_processLogs) (108588) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:16,162::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:16,163::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:16,164::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:16,166::__init__::232::IOProcess::(_processLogs) (108589) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:16,167::__init__::232::IOProcess::(_processLogs) (108589) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:16,170::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:16,171::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:16,172::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:16,173::__init__::232::IOProcess::(_processLogs) (108590) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:16,175::__init__::232::IOProcess::(_processLogs) (108590) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:16,177::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:16,178::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:16,179::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:16,181::__init__::232::IOProcess::(_processLogs) (108591) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:16,182::__init__::232::IOProcess::(_processLogs) (108591) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:16,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:17,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185965 s, 55.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:18,299::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:18,301::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70586::DEBUG::2014-09-25 17:33:18,322::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:19,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:19,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0266598 s, 38.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:33:21,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:21,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133394 s, 76.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:21,112::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:21,113::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:21,114::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:21,114::__init__::232::IOProcess::(_processLogs) (108592) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:21,116::__init__::232::IOProcess::(_processLogs) (108592) Queuing response >Thread-81::DEBUG::2014-09-25 17:33:21,117::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:33:21,131::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00160295 s, 231 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:21,133::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:21,134::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:21,136::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:21,138::__init__::232::IOProcess::(_processLogs) (108593) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:21,139::__init__::232::IOProcess::(_processLogs) (108593) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:21,335::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:21,336::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70587::DEBUG::2014-09-25 17:33:21,338::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:23,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:23,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133635 s, 76.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:23,699::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:23,700::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70588::DEBUG::2014-09-25 17:33:23,700::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70588::DEBUG::2014-09-25 17:33:23,702::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b9fe20f-07ce-47fc-a8b6-9ae55776f32e`::moving from state init -> state preparing >Thread-70588::INFO::2014-09-25 17:33:23,702::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70588::INFO::2014-09-25 17:33:23,703::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70588::DEBUG::2014-09-25 17:33:23,703::task::1191::Storage.TaskManager.Task::(prepare) Task=`3b9fe20f-07ce-47fc-a8b6-9ae55776f32e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70588::DEBUG::2014-09-25 17:33:23,703::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b9fe20f-07ce-47fc-a8b6-9ae55776f32e`::moving from state preparing -> state finished >Thread-70588::DEBUG::2014-09-25 17:33:23,704::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70588::DEBUG::2014-09-25 17:33:23,704::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70588::DEBUG::2014-09-25 17:33:23,704::task::993::Storage.TaskManager.Task::(_decref) Task=`3b9fe20f-07ce-47fc-a8b6-9ae55776f32e`::ref 0 aborting False >Thread-70588::DEBUG::2014-09-25 17:33:23,704::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70588::DEBUG::2014-09-25 17:33:23,705::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:23,719::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70589::DEBUG::2014-09-25 17:33:23,720::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:33:23,720::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70589::DEBUG::2014-09-25 17:33:23,722::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ac763e5-70b2-4034-8d13-ac6ff0cf0e39`::moving from state init -> state preparing >Thread-70589::INFO::2014-09-25 17:33:23,722::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70589::DEBUG::2014-09-25 17:33:23,723::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e1e97711-e7b8-48ac-b07c-b12603433afc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70589::DEBUG::2014-09-25 17:33:23,723::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70589::DEBUG::2014-09-25 17:33:23,724::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70589::DEBUG::2014-09-25 17:33:23,724::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e1e97711-e7b8-48ac-b07c-b12603433afc`::Granted request >Thread-70589::DEBUG::2014-09-25 17:33:23,724::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`8ac763e5-70b2-4034-8d13-ac6ff0cf0e39`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70589::DEBUG::2014-09-25 17:33:23,725::task::993::Storage.TaskManager.Task::(_decref) Task=`8ac763e5-70b2-4034-8d13-ac6ff0cf0e39`::ref 1 aborting False >Thread-70589::INFO::2014-09-25 17:33:23,729::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70589::DEBUG::2014-09-25 17:33:23,729::task::1191::Storage.TaskManager.Task::(prepare) Task=`8ac763e5-70b2-4034-8d13-ac6ff0cf0e39`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70589::DEBUG::2014-09-25 17:33:23,729::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ac763e5-70b2-4034-8d13-ac6ff0cf0e39`::moving from state preparing -> state finished >Thread-70589::DEBUG::2014-09-25 17:33:23,730::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70589::DEBUG::2014-09-25 17:33:23,730::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70589::DEBUG::2014-09-25 17:33:23,730::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70589::DEBUG::2014-09-25 17:33:23,731::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70589::DEBUG::2014-09-25 17:33:23,731::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70589::DEBUG::2014-09-25 17:33:23,731::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70589::DEBUG::2014-09-25 17:33:23,731::task::993::Storage.TaskManager.Task::(_decref) Task=`8ac763e5-70b2-4034-8d13-ac6ff0cf0e39`::ref 0 aborting False >Thread-70589::DEBUG::2014-09-25 17:33:23,732::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70589::DEBUG::2014-09-25 17:33:23,732::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:24,350::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:24,351::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70590::DEBUG::2014-09-25 17:33:24,353::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:33:24,700::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:24,701::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:24,701::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:24,701::__init__::232::IOProcess::(_processLogs) (108594) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:24,701::__init__::232::IOProcess::(_processLogs) (108594) Queuing response >Thread-80::DEBUG::2014-09-25 17:33:24,703::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:33:24,712::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000464376 s, 814 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:24,713::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:24,713::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:24,713::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:24,713::__init__::232::IOProcess::(_processLogs) (108595) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:24,714::__init__::232::IOProcess::(_processLogs) (108595) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:25,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:25,220::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144214 s, 71.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:26,182::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:26,182::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:26,182::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:26,183::__init__::232::IOProcess::(_processLogs) (108596) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:26,185::__init__::232::IOProcess::(_processLogs) (108596) Queuing response >Thread-79::DEBUG::2014-09-25 17:33:26,186::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:33:26,201::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00468339 s, 120 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:26,202::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:26,202::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:26,202::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:26,203::__init__::232::IOProcess::(_processLogs) (108597) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:26,205::__init__::232::IOProcess::(_processLogs) (108597) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:26,206::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:26,207::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:26,207::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:26,208::__init__::232::IOProcess::(_processLogs) (108598) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:26,208::__init__::232::IOProcess::(_processLogs) (108598) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:26,208::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:26,209::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:26,209::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:26,209::__init__::232::IOProcess::(_processLogs) (108599) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:26,209::__init__::232::IOProcess::(_processLogs) (108599) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:26,210::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:26,210::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:26,211::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:26,211::__init__::232::IOProcess::(_processLogs) (108600) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:26,211::__init__::232::IOProcess::(_processLogs) (108600) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:27,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:27,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135247 s, 75.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:27,364::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:27,366::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70591::DEBUG::2014-09-25 17:33:27,368::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:29,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:29,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210237 s, 48.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:30,382::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:30,384::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70592::DEBUG::2014-09-25 17:33:30,387::task::595::Storage.TaskManager.Task::(_updateState) Task=`e6976934-bb93-4e05-9e00-9135833794fa`::moving from state init -> state preparing >Thread-70592::INFO::2014-09-25 17:33:30,387::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70592::INFO::2014-09-25 17:33:30,387::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00468339', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000464376', 'lastCheck': '5.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00160295', 'lastCheck': '9.2', 'valid': True}} >Thread-70592::DEBUG::2014-09-25 17:33:30,388::task::1191::Storage.TaskManager.Task::(prepare) Task=`e6976934-bb93-4e05-9e00-9135833794fa`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00468339', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000464376', 'lastCheck': '5.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00160295', 'lastCheck': '9.2', 'valid': True}} >Thread-70592::DEBUG::2014-09-25 17:33:30,388::task::595::Storage.TaskManager.Task::(_updateState) Task=`e6976934-bb93-4e05-9e00-9135833794fa`::moving from state preparing -> state finished >Thread-70592::DEBUG::2014-09-25 17:33:30,388::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70592::DEBUG::2014-09-25 17:33:30,388::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70592::DEBUG::2014-09-25 17:33:30,388::task::993::Storage.TaskManager.Task::(_decref) Task=`e6976934-bb93-4e05-9e00-9135833794fa`::ref 0 aborting False >Thread-70592::DEBUG::2014-09-25 17:33:30,391::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:30,412::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:30,413::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70593::DEBUG::2014-09-25 17:33:30,415::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:33:31,139::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:31,139::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:31,139::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:31,139::__init__::232::IOProcess::(_processLogs) (108601) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:31,143::__init__::232::IOProcess::(_processLogs) (108601) Queuing response >Thread-81::DEBUG::2014-09-25 17:33:31,144::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:33:31,158::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00372117 s, 99.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:31,160::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:31,160::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:31,160::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:31,160::__init__::232::IOProcess::(_processLogs) (108602) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:31,164::__init__::232::IOProcess::(_processLogs) (108602) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:31,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:31,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161702 s, 63.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:33:33,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:33,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131565 s, 77.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:33,469::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:33,471::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70594::DEBUG::2014-09-25 17:33:33,473::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:33,765::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:33,767::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70595::DEBUG::2014-09-25 17:33:33,767::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70595::DEBUG::2014-09-25 17:33:33,769::task::595::Storage.TaskManager.Task::(_updateState) Task=`b52f1d39-232e-4764-9f72-bb8798b62255`::moving from state init -> state preparing >Thread-70595::INFO::2014-09-25 17:33:33,769::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70595::INFO::2014-09-25 17:33:33,770::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70595::DEBUG::2014-09-25 17:33:33,770::task::1191::Storage.TaskManager.Task::(prepare) Task=`b52f1d39-232e-4764-9f72-bb8798b62255`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70595::DEBUG::2014-09-25 17:33:33,770::task::595::Storage.TaskManager.Task::(_updateState) Task=`b52f1d39-232e-4764-9f72-bb8798b62255`::moving from state preparing -> state finished >Thread-70595::DEBUG::2014-09-25 17:33:33,770::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70595::DEBUG::2014-09-25 17:33:33,770::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70595::DEBUG::2014-09-25 17:33:33,771::task::993::Storage.TaskManager.Task::(_decref) Task=`b52f1d39-232e-4764-9f72-bb8798b62255`::ref 0 aborting False >Thread-70595::DEBUG::2014-09-25 17:33:33,771::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70595::DEBUG::2014-09-25 17:33:33,771::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:33,785::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:33,787::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70596::DEBUG::2014-09-25 17:33:33,787::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70596::DEBUG::2014-09-25 17:33:33,788::task::595::Storage.TaskManager.Task::(_updateState) Task=`58460f03-004d-4a08-902f-8125ccd8e6b5`::moving from state init -> state preparing >Thread-70596::INFO::2014-09-25 17:33:33,789::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70596::DEBUG::2014-09-25 17:33:33,789::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4aecbd82-02d6-43cb-afb5-effefff3fb1c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70596::DEBUG::2014-09-25 17:33:33,789::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70596::DEBUG::2014-09-25 17:33:33,790::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70596::DEBUG::2014-09-25 17:33:33,790::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4aecbd82-02d6-43cb-afb5-effefff3fb1c`::Granted request >Thread-70596::DEBUG::2014-09-25 17:33:33,791::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`58460f03-004d-4a08-902f-8125ccd8e6b5`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70596::DEBUG::2014-09-25 17:33:33,791::task::993::Storage.TaskManager.Task::(_decref) Task=`58460f03-004d-4a08-902f-8125ccd8e6b5`::ref 1 aborting False >Thread-70596::INFO::2014-09-25 17:33:33,795::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70596::DEBUG::2014-09-25 17:33:33,796::task::1191::Storage.TaskManager.Task::(prepare) Task=`58460f03-004d-4a08-902f-8125ccd8e6b5`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70596::DEBUG::2014-09-25 17:33:33,796::task::595::Storage.TaskManager.Task::(_updateState) Task=`58460f03-004d-4a08-902f-8125ccd8e6b5`::moving from state preparing -> state finished >Thread-70596::DEBUG::2014-09-25 17:33:33,796::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70596::DEBUG::2014-09-25 17:33:33,796::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70596::DEBUG::2014-09-25 17:33:33,797::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70596::DEBUG::2014-09-25 17:33:33,797::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70596::DEBUG::2014-09-25 17:33:33,797::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70596::DEBUG::2014-09-25 17:33:33,798::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70596::DEBUG::2014-09-25 17:33:33,798::task::993::Storage.TaskManager.Task::(_decref) Task=`58460f03-004d-4a08-902f-8125ccd8e6b5`::ref 0 aborting False >Thread-70596::DEBUG::2014-09-25 17:33:33,798::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70596::DEBUG::2014-09-25 17:33:33,799::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:33:34,716::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:34,717::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:34,717::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:34,717::__init__::232::IOProcess::(_processLogs) (108603) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:34,717::__init__::232::IOProcess::(_processLogs) (108603) Queuing response >Thread-80::DEBUG::2014-09-25 17:33:34,719::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:33:34,727::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000365633 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:34,729::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:34,729::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:34,729::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:34,731::__init__::232::IOProcess::(_processLogs) (108604) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:34,731::__init__::232::IOProcess::(_processLogs) (108604) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:35,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:35,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134817 s, 76.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:36,213::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:36,213::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:36,213::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:36,214::__init__::232::IOProcess::(_processLogs) (108605) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:36,226::__init__::232::IOProcess::(_processLogs) (108605) Queuing response >Thread-79::DEBUG::2014-09-25 17:33:36,227::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:33:36,250::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00259519 s, 217 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:36,251::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:36,251::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:36,251::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:36,251::__init__::232::IOProcess::(_processLogs) (108606) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:36,261::__init__::232::IOProcess::(_processLogs) (108606) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:36,262::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:36,263::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:36,263::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:36,263::__init__::232::IOProcess::(_processLogs) (108607) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:36,263::__init__::232::IOProcess::(_processLogs) (108607) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:36,264::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:36,265::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:36,265::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:36,266::__init__::232::IOProcess::(_processLogs) (108608) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:36,266::__init__::232::IOProcess::(_processLogs) (108608) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:36,266::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:36,267::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:36,267::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:36,267::__init__::232::IOProcess::(_processLogs) (108609) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:36,267::__init__::232::IOProcess::(_processLogs) (108609) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:36,491::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:36,492::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70597::DEBUG::2014-09-25 17:33:36,494::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:37,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:37,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138016 s, 74.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:33:39,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:39,517::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:39,519::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70598::DEBUG::2014-09-25 17:33:39,521::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:39,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181323 s, 56.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:41,166::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:41,167::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:41,167::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:41,167::__init__::232::IOProcess::(_processLogs) (108610) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:41,168::__init__::232::IOProcess::(_processLogs) (108610) Queuing response >Thread-81::DEBUG::2014-09-25 17:33:41,169::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:33:41,187::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00286041 s, 130 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:41,188::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:41,188::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:41,189::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:41,189::__init__::232::IOProcess::(_processLogs) (108611) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:41,193::__init__::232::IOProcess::(_processLogs) (108611) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:41,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:41,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133022 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:42,539::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:42,540::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70599::DEBUG::2014-09-25 17:33:42,543::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:43,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:43,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150005 s, 68.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:43,828::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70600::DEBUG::2014-09-25 17:33:43,830::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:33:43,830::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70600::DEBUG::2014-09-25 17:33:43,832::task::595::Storage.TaskManager.Task::(_updateState) Task=`a4c4be89-b9ae-4a06-a78c-e06699e8e317`::moving from state init -> state preparing >Thread-70600::INFO::2014-09-25 17:33:43,833::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70600::INFO::2014-09-25 17:33:43,833::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70600::DEBUG::2014-09-25 17:33:43,834::task::1191::Storage.TaskManager.Task::(prepare) Task=`a4c4be89-b9ae-4a06-a78c-e06699e8e317`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70600::DEBUG::2014-09-25 17:33:43,834::task::595::Storage.TaskManager.Task::(_updateState) Task=`a4c4be89-b9ae-4a06-a78c-e06699e8e317`::moving from state preparing -> state finished >Thread-70600::DEBUG::2014-09-25 17:33:43,834::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70600::DEBUG::2014-09-25 17:33:43,834::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70600::DEBUG::2014-09-25 17:33:43,834::task::993::Storage.TaskManager.Task::(_decref) Task=`a4c4be89-b9ae-4a06-a78c-e06699e8e317`::ref 0 aborting False >Thread-70600::DEBUG::2014-09-25 17:33:43,835::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70600::DEBUG::2014-09-25 17:33:43,835::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:43,846::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:43,847::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70601::DEBUG::2014-09-25 17:33:43,847::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70601::DEBUG::2014-09-25 17:33:43,849::task::595::Storage.TaskManager.Task::(_updateState) Task=`c7120f59-7e03-47c1-a33c-8670cbd81d6d`::moving from state init -> state preparing >Thread-70601::INFO::2014-09-25 17:33:43,849::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70601::DEBUG::2014-09-25 17:33:43,850::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9eba6d23-def2-4454-b152-33a8d46890e4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70601::DEBUG::2014-09-25 17:33:43,850::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70601::DEBUG::2014-09-25 17:33:43,851::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70601::DEBUG::2014-09-25 17:33:43,851::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9eba6d23-def2-4454-b152-33a8d46890e4`::Granted request >Thread-70601::DEBUG::2014-09-25 17:33:43,851::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c7120f59-7e03-47c1-a33c-8670cbd81d6d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70601::DEBUG::2014-09-25 17:33:43,852::task::993::Storage.TaskManager.Task::(_decref) Task=`c7120f59-7e03-47c1-a33c-8670cbd81d6d`::ref 1 aborting False >Thread-70601::INFO::2014-09-25 17:33:43,856::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70601::DEBUG::2014-09-25 17:33:43,856::task::1191::Storage.TaskManager.Task::(prepare) Task=`c7120f59-7e03-47c1-a33c-8670cbd81d6d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70601::DEBUG::2014-09-25 17:33:43,857::task::595::Storage.TaskManager.Task::(_updateState) Task=`c7120f59-7e03-47c1-a33c-8670cbd81d6d`::moving from state preparing -> state finished >Thread-70601::DEBUG::2014-09-25 17:33:43,857::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70601::DEBUG::2014-09-25 17:33:43,857::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70601::DEBUG::2014-09-25 17:33:43,857::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70601::DEBUG::2014-09-25 17:33:43,858::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70601::DEBUG::2014-09-25 17:33:43,858::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70601::DEBUG::2014-09-25 17:33:43,858::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70601::DEBUG::2014-09-25 17:33:43,859::task::993::Storage.TaskManager.Task::(_decref) Task=`c7120f59-7e03-47c1-a33c-8670cbd81d6d`::ref 0 aborting False >Thread-70601::DEBUG::2014-09-25 17:33:43,859::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70601::DEBUG::2014-09-25 17:33:43,860::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:33:44,732::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:44,732::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:44,732::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:44,733::__init__::232::IOProcess::(_processLogs) (108612) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:44,733::__init__::232::IOProcess::(_processLogs) (108612) Queuing response >Thread-80::DEBUG::2014-09-25 17:33:44,735::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:33:44,744::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000353477 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:44,745::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:44,745::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:44,746::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:44,746::__init__::232::IOProcess::(_processLogs) (108613) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:44,746::__init__::232::IOProcess::(_processLogs) (108613) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:45,559::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:45,560::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70602::DEBUG::2014-09-25 17:33:45,563::task::595::Storage.TaskManager.Task::(_updateState) Task=`1fdf5838-1f87-401f-a8ae-75375f5901e7`::moving from state init -> state preparing >Thread-70602::INFO::2014-09-25 17:33:45,564::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70602::INFO::2014-09-25 17:33:45,564::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00259519', 'lastCheck': '9.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000353477', 'lastCheck': '0.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00286041', 'lastCheck': '4.4', 'valid': True}} >Thread-70602::DEBUG::2014-09-25 17:33:45,565::task::1191::Storage.TaskManager.Task::(prepare) Task=`1fdf5838-1f87-401f-a8ae-75375f5901e7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00259519', 'lastCheck': '9.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000353477', 'lastCheck': '0.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00286041', 'lastCheck': '4.4', 'valid': True}} >Thread-70602::DEBUG::2014-09-25 17:33:45,565::task::595::Storage.TaskManager.Task::(_updateState) Task=`1fdf5838-1f87-401f-a8ae-75375f5901e7`::moving from state preparing -> state finished >Thread-70602::DEBUG::2014-09-25 17:33:45,565::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70602::DEBUG::2014-09-25 17:33:45,565::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70602::DEBUG::2014-09-25 17:33:45,566::task::993::Storage.TaskManager.Task::(_decref) Task=`1fdf5838-1f87-401f-a8ae-75375f5901e7`::ref 0 aborting False >Thread-70602::DEBUG::2014-09-25 17:33:45,568::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:45,590::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:45,591::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70603::DEBUG::2014-09-25 17:33:45,593::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:45,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:45,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182578 s, 56.1 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 17:33:46,268::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:33:46,275::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,277::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,279::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,281::__init__::232::IOProcess::(_processLogs) (108614) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:33:46,285::__init__::232::IOProcess::(_processLogs) (108614) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:46,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,290::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,292::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,294::__init__::232::IOProcess::(_processLogs) (108615) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:33:46,295::__init__::232::IOProcess::(_processLogs) (108615) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:46,300::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,304::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,308::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,313::__init__::232::IOProcess::(_processLogs) (108616) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:33:46,317::__init__::232::IOProcess::(_processLogs) (108616) Queuing response >Thread-79::DEBUG::2014-09-25 17:33:46,321::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:33:46,324::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,326::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,327::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,329::__init__::232::IOProcess::(_processLogs) (108617) Got request for method 'touch' >Thread-79::DEBUG::2014-09-25 17:33:46,374::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:33:46,374::__init__::232::IOProcess::(_processLogs) (108617) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:46,378::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,380::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,382::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,383::__init__::232::IOProcess::(_processLogs) (108618) Got request for method 'readfile' >Thread-79::DEBUG::2014-09-25 17:33:46,386::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:33:46,386::__init__::232::IOProcess::(_processLogs) (108618) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:46,391::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,393::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,395::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,397::__init__::232::IOProcess::(_processLogs) (108619) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:46,399::__init__::232::IOProcess::(_processLogs) (108619) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:46,402::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,404::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,406::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,408::__init__::232::IOProcess::(_processLogs) (108620) Got request for method 'glob' >Thread-79::DEBUG::2014-09-25 17:33:46,410::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:33:46,411::__init__::232::IOProcess::(_processLogs) (108620) Queuing response >Thread-79::INFO::2014-09-25 17:33:46,412::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:33:46,414::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:33:46,416::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,418::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,420::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,422::__init__::232::IOProcess::(_processLogs) (108621) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:46,425::__init__::232::IOProcess::(_processLogs) (108621) Queuing response >Thread-79::DEBUG::2014-09-25 17:33:46,426::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:33:46,457::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0118912 s, 47.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:46,460::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,461::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,462::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,464::__init__::232::IOProcess::(_processLogs) (108622) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:46,465::__init__::232::IOProcess::(_processLogs) (108622) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:46,468::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,469::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,470::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,471::__init__::232::IOProcess::(_processLogs) (108623) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:46,473::__init__::232::IOProcess::(_processLogs) (108623) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:46,475::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,476::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,477::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,478::__init__::232::IOProcess::(_processLogs) (108624) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:46,480::__init__::232::IOProcess::(_processLogs) (108624) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:46,482::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:46,483::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:46,484::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:46,485::__init__::232::IOProcess::(_processLogs) (108625) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:46,487::__init__::232::IOProcess::(_processLogs) (108625) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:47,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:47,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0223588 s, 45.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:48,647::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:48,648::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70607::DEBUG::2014-09-25 17:33:48,650::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:49,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:49,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207975 s, 49.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:51,196::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:51,197::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-81::DEBUG::2014-09-25 17:33:51,198::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:33:51,198::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:51,206::__init__::232::IOProcess::(_processLogs) (108626) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:51,207::__init__::232::IOProcess::(_processLogs) (108626) Queuing response >Thread-81::DEBUG::2014-09-25 17:33:51,209::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00148063 s, 251 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:51,210::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:51,210::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:51,210::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:51,211::__init__::232::IOProcess::(_processLogs) (108627) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:51,212::__init__::232::IOProcess::(_processLogs) (108627) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:51,665::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:51,667::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70608::DEBUG::2014-09-25 17:33:51,669::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:51,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:52,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0305475 s, 33.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:53,884::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:53,886::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70609::DEBUG::2014-09-25 17:33:53,886::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70609::DEBUG::2014-09-25 17:33:53,888::task::595::Storage.TaskManager.Task::(_updateState) Task=`93a3e9dc-af7d-48d8-b14e-9724fc64d5f0`::moving from state init -> state preparing >Thread-70609::INFO::2014-09-25 17:33:53,888::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70609::INFO::2014-09-25 17:33:53,889::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70609::DEBUG::2014-09-25 17:33:53,889::task::1191::Storage.TaskManager.Task::(prepare) Task=`93a3e9dc-af7d-48d8-b14e-9724fc64d5f0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70609::DEBUG::2014-09-25 17:33:53,889::task::595::Storage.TaskManager.Task::(_updateState) Task=`93a3e9dc-af7d-48d8-b14e-9724fc64d5f0`::moving from state preparing -> state finished >Thread-70609::DEBUG::2014-09-25 17:33:53,889::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70609::DEBUG::2014-09-25 17:33:53,889::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70609::DEBUG::2014-09-25 17:33:53,890::task::993::Storage.TaskManager.Task::(_decref) Task=`93a3e9dc-af7d-48d8-b14e-9724fc64d5f0`::ref 0 aborting False >Thread-70609::DEBUG::2014-09-25 17:33:53,890::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70609::DEBUG::2014-09-25 17:33:53,890::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:53,903::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:53,905::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70610::DEBUG::2014-09-25 17:33:53,905::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70610::DEBUG::2014-09-25 17:33:53,906::task::595::Storage.TaskManager.Task::(_updateState) Task=`e607892a-0fed-4ea7-a625-5b495e532642`::moving from state init -> state preparing >Thread-70610::INFO::2014-09-25 17:33:53,907::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70610::DEBUG::2014-09-25 17:33:53,907::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ca3f681c-d34f-4413-8bec-e7e8c8b05a75`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70610::DEBUG::2014-09-25 17:33:53,908::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70610::DEBUG::2014-09-25 17:33:53,908::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70610::DEBUG::2014-09-25 17:33:53,908::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ca3f681c-d34f-4413-8bec-e7e8c8b05a75`::Granted request >Thread-70610::DEBUG::2014-09-25 17:33:53,909::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e607892a-0fed-4ea7-a625-5b495e532642`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70610::DEBUG::2014-09-25 17:33:53,909::task::993::Storage.TaskManager.Task::(_decref) Task=`e607892a-0fed-4ea7-a625-5b495e532642`::ref 1 aborting False >Thread-70610::INFO::2014-09-25 17:33:53,913::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70610::DEBUG::2014-09-25 17:33:53,914::task::1191::Storage.TaskManager.Task::(prepare) Task=`e607892a-0fed-4ea7-a625-5b495e532642`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70610::DEBUG::2014-09-25 17:33:53,914::task::595::Storage.TaskManager.Task::(_updateState) Task=`e607892a-0fed-4ea7-a625-5b495e532642`::moving from state preparing -> state finished >Thread-70610::DEBUG::2014-09-25 17:33:53,914::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70610::DEBUG::2014-09-25 17:33:53,914::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70610::DEBUG::2014-09-25 17:33:53,915::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70610::DEBUG::2014-09-25 17:33:53,915::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70610::DEBUG::2014-09-25 17:33:53,915::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70610::DEBUG::2014-09-25 17:33:53,915::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70610::DEBUG::2014-09-25 17:33:53,916::task::993::Storage.TaskManager.Task::(_decref) Task=`e607892a-0fed-4ea7-a625-5b495e532642`::ref 0 aborting False >Thread-70610::DEBUG::2014-09-25 17:33:53,916::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70610::DEBUG::2014-09-25 17:33:53,917::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:54,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:54,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148094 s, 69.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:54,684::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:54,686::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70611::DEBUG::2014-09-25 17:33:54,688::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:33:54,749::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:54,749::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:54,750::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:33:54,751::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:33:54,751::__init__::232::IOProcess::(_processLogs) (108628) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:54,759::__init__::232::IOProcess::(_processLogs) (108628) Queuing response >Thread-80::DEBUG::2014-09-25 17:33:54,760::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000396382 s, 954 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:54,761::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:54,762::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:54,762::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:54,764::__init__::232::IOProcess::(_processLogs) (108629) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:54,764::__init__::232::IOProcess::(_processLogs) (108629) Queuing response >Dummy-148::DEBUG::2014-09-25 17:33:56,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:56,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158166 s, 64.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:56,489::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:56,490::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:56,490::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:56,490::__init__::232::IOProcess::(_processLogs) (108630) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:56,496::__init__::232::IOProcess::(_processLogs) (108630) Queuing response >Thread-79::DEBUG::2014-09-25 17:33:56,497::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:33:56,523::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0164128 s, 34.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:33:56,525::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:56,525::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:56,525::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:56,525::__init__::232::IOProcess::(_processLogs) (108631) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:33:56,534::__init__::232::IOProcess::(_processLogs) (108631) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:56,535::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:56,536::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:56,536::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:56,537::__init__::232::IOProcess::(_processLogs) (108632) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:56,537::__init__::232::IOProcess::(_processLogs) (108632) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:56,537::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:56,538::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:56,538::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:56,539::__init__::232::IOProcess::(_processLogs) (108633) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:56,539::__init__::232::IOProcess::(_processLogs) (108633) Queuing response >Thread-29::DEBUG::2014-09-25 17:33:56,540::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:33:56,540::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:33:56,540::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:33:56,541::__init__::232::IOProcess::(_processLogs) (108634) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:33:56,542::__init__::232::IOProcess::(_processLogs) (108634) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:33:57,702::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:33:57,704::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70612::DEBUG::2014-09-25 17:33:57,706::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:33:58,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:33:58,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132811 s, 77.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:34:00,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:00,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171664 s, 59.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:00,721::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:00,723::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70613::DEBUG::2014-09-25 17:34:00,725::task::595::Storage.TaskManager.Task::(_updateState) Task=`6de6fee3-825b-4b71-9721-a18b733f16ba`::moving from state init -> state preparing >Thread-70613::INFO::2014-09-25 17:34:00,726::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70613::INFO::2014-09-25 17:34:00,726::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0164128', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000396382', 'lastCheck': '6.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00148063', 'lastCheck': '9.5', 'valid': True}} >Thread-70613::DEBUG::2014-09-25 17:34:00,727::task::1191::Storage.TaskManager.Task::(prepare) Task=`6de6fee3-825b-4b71-9721-a18b733f16ba`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0164128', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000396382', 'lastCheck': '6.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00148063', 'lastCheck': '9.5', 'valid': True}} >Thread-70613::DEBUG::2014-09-25 17:34:00,727::task::595::Storage.TaskManager.Task::(_updateState) Task=`6de6fee3-825b-4b71-9721-a18b733f16ba`::moving from state preparing -> state finished >Thread-70613::DEBUG::2014-09-25 17:34:00,727::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70613::DEBUG::2014-09-25 17:34:00,727::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70613::DEBUG::2014-09-25 17:34:00,727::task::993::Storage.TaskManager.Task::(_decref) Task=`6de6fee3-825b-4b71-9721-a18b733f16ba`::ref 0 aborting False >Thread-70613::DEBUG::2014-09-25 17:34:00,730::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:00,755::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:00,756::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70614::DEBUG::2014-09-25 17:34:00,758::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:34:01,214::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:01,214::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:01,214::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:01,215::__init__::232::IOProcess::(_processLogs) (108635) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:01,240::__init__::232::IOProcess::(_processLogs) (108635) Queuing response >Thread-81::DEBUG::2014-09-25 17:34:01,241::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:34:01,265::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00328829 s, 113 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:01,266::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:01,267::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:01,267::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:01,267::__init__::232::IOProcess::(_processLogs) (108636) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:01,272::__init__::232::IOProcess::(_processLogs) (108636) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:02,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:02,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015283 s, 67.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:03,819::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:03,821::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70615::DEBUG::2014-09-25 17:34:03,823::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:03,952::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70616::DEBUG::2014-09-25 17:34:03,953::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:34:03,953::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70616::DEBUG::2014-09-25 17:34:03,955::task::595::Storage.TaskManager.Task::(_updateState) Task=`149458bf-881a-47df-954d-b4efc059ac8b`::moving from state init -> state preparing >Thread-70616::INFO::2014-09-25 17:34:03,955::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70616::INFO::2014-09-25 17:34:03,956::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70616::DEBUG::2014-09-25 17:34:03,956::task::1191::Storage.TaskManager.Task::(prepare) Task=`149458bf-881a-47df-954d-b4efc059ac8b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70616::DEBUG::2014-09-25 17:34:03,956::task::595::Storage.TaskManager.Task::(_updateState) Task=`149458bf-881a-47df-954d-b4efc059ac8b`::moving from state preparing -> state finished >Thread-70616::DEBUG::2014-09-25 17:34:03,957::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70616::DEBUG::2014-09-25 17:34:03,957::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70616::DEBUG::2014-09-25 17:34:03,957::task::993::Storage.TaskManager.Task::(_decref) Task=`149458bf-881a-47df-954d-b4efc059ac8b`::ref 0 aborting False >Thread-70616::DEBUG::2014-09-25 17:34:03,957::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70616::DEBUG::2014-09-25 17:34:03,958::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:03,968::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70617::DEBUG::2014-09-25 17:34:03,969::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:34:03,969::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70617::DEBUG::2014-09-25 17:34:03,971::task::595::Storage.TaskManager.Task::(_updateState) Task=`46c36622-1fea-41da-aaed-16190b0935b5`::moving from state init -> state preparing >Thread-70617::INFO::2014-09-25 17:34:03,972::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70617::DEBUG::2014-09-25 17:34:03,972::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b3020c8d-175e-4bbd-a755-3a8f1b7ad4ea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70617::DEBUG::2014-09-25 17:34:03,972::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70617::DEBUG::2014-09-25 17:34:03,973::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70617::DEBUG::2014-09-25 17:34:03,973::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b3020c8d-175e-4bbd-a755-3a8f1b7ad4ea`::Granted request >Thread-70617::DEBUG::2014-09-25 17:34:03,974::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`46c36622-1fea-41da-aaed-16190b0935b5`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70617::DEBUG::2014-09-25 17:34:03,974::task::993::Storage.TaskManager.Task::(_decref) Task=`46c36622-1fea-41da-aaed-16190b0935b5`::ref 1 aborting False >Thread-70617::INFO::2014-09-25 17:34:03,978::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7248805888', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70617::DEBUG::2014-09-25 17:34:03,979::task::1191::Storage.TaskManager.Task::(prepare) Task=`46c36622-1fea-41da-aaed-16190b0935b5`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7248805888', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70617::DEBUG::2014-09-25 17:34:03,979::task::595::Storage.TaskManager.Task::(_updateState) Task=`46c36622-1fea-41da-aaed-16190b0935b5`::moving from state preparing -> state finished >Thread-70617::DEBUG::2014-09-25 17:34:03,979::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70617::DEBUG::2014-09-25 17:34:03,979::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70617::DEBUG::2014-09-25 17:34:03,980::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70617::DEBUG::2014-09-25 17:34:03,980::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70617::DEBUG::2014-09-25 17:34:03,980::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70617::DEBUG::2014-09-25 17:34:03,980::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70617::DEBUG::2014-09-25 17:34:03,981::task::993::Storage.TaskManager.Task::(_decref) Task=`46c36622-1fea-41da-aaed-16190b0935b5`::ref 0 aborting False >Thread-70617::DEBUG::2014-09-25 17:34:03,981::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7248805888', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70617::DEBUG::2014-09-25 17:34:03,982::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:04,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:04,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165621 s, 61.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:04,764::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:04,765::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:04,765::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:04,765::__init__::232::IOProcess::(_processLogs) (108637) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:04,766::__init__::232::IOProcess::(_processLogs) (108637) Queuing response >Thread-80::DEBUG::2014-09-25 17:34:04,767::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:34:04,776::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000455826 s, 829 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:04,777::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:04,778::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:04,778::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:04,780::__init__::232::IOProcess::(_processLogs) (108638) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:04,780::__init__::232::IOProcess::(_processLogs) (108638) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:06,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:06,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132422 s, 77.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:06,542::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:06,543::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:06,543::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:34:06,544::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:34:06,545::__init__::232::IOProcess::(_processLogs) (108639) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:06,553::__init__::232::IOProcess::(_processLogs) (108639) Queuing response >Thread-79::DEBUG::2014-09-25 17:34:06,554::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000489121 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:06,555::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:06,555::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:06,556::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:06,556::__init__::232::IOProcess::(_processLogs) (108640) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:06,556::__init__::232::IOProcess::(_processLogs) (108640) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:06,558::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:06,558::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:06,559::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:06,559::__init__::232::IOProcess::(_processLogs) (108641) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:06,559::__init__::232::IOProcess::(_processLogs) (108641) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:06,560::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:06,560::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:06,561::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:06,561::__init__::232::IOProcess::(_processLogs) (108642) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:06,561::__init__::232::IOProcess::(_processLogs) (108642) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:06,562::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:06,562::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:06,563::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:06,564::__init__::232::IOProcess::(_processLogs) (108643) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:06,564::__init__::232::IOProcess::(_processLogs) (108643) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:06,839::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:06,840::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70618::DEBUG::2014-09-25 17:34:06,842::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:08,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:08,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0176267 s, 58.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:09,859::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:09,860::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70619::DEBUG::2014-09-25 17:34:09,862::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:10,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:10,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154306 s, 66.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:11,274::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:11,274::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:11,275::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:11,275::__init__::232::IOProcess::(_processLogs) (108644) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:11,277::__init__::232::IOProcess::(_processLogs) (108644) Queuing response >Thread-81::DEBUG::2014-09-25 17:34:11,278::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:34:11,322::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0278208 s, 13.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:11,324::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:11,324::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:11,324::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:11,324::__init__::232::IOProcess::(_processLogs) (108645) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:11,337::__init__::232::IOProcess::(_processLogs) (108645) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:12,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:12,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150719 s, 67.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:12,874::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:12,876::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70620::DEBUG::2014-09-25 17:34:12,878::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:14,003::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:14,005::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70621::DEBUG::2014-09-25 17:34:14,005::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70621::DEBUG::2014-09-25 17:34:14,007::task::595::Storage.TaskManager.Task::(_updateState) Task=`403c2f97-5d86-4447-a5b1-4a31902b59a7`::moving from state init -> state preparing >Thread-70621::INFO::2014-09-25 17:34:14,007::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70621::INFO::2014-09-25 17:34:14,008::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70621::DEBUG::2014-09-25 17:34:14,008::task::1191::Storage.TaskManager.Task::(prepare) Task=`403c2f97-5d86-4447-a5b1-4a31902b59a7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70621::DEBUG::2014-09-25 17:34:14,008::task::595::Storage.TaskManager.Task::(_updateState) Task=`403c2f97-5d86-4447-a5b1-4a31902b59a7`::moving from state preparing -> state finished >Thread-70621::DEBUG::2014-09-25 17:34:14,008::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70621::DEBUG::2014-09-25 17:34:14,009::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70621::DEBUG::2014-09-25 17:34:14,009::task::993::Storage.TaskManager.Task::(_decref) Task=`403c2f97-5d86-4447-a5b1-4a31902b59a7`::ref 0 aborting False >Thread-70621::DEBUG::2014-09-25 17:34:14,009::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70621::DEBUG::2014-09-25 17:34:14,010::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:14,020::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:14,021::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70622::DEBUG::2014-09-25 17:34:14,021::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70622::DEBUG::2014-09-25 17:34:14,023::task::595::Storage.TaskManager.Task::(_updateState) Task=`07cbef13-faf6-4f55-9130-e4fed9c4eacf`::moving from state init -> state preparing >Thread-70622::INFO::2014-09-25 17:34:14,023::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70622::DEBUG::2014-09-25 17:34:14,024::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3798931e-2582-4b4d-835c-467b61e7305b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70622::DEBUG::2014-09-25 17:34:14,024::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70622::DEBUG::2014-09-25 17:34:14,025::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70622::DEBUG::2014-09-25 17:34:14,025::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3798931e-2582-4b4d-835c-467b61e7305b`::Granted request >Thread-70622::DEBUG::2014-09-25 17:34:14,026::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`07cbef13-faf6-4f55-9130-e4fed9c4eacf`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70622::DEBUG::2014-09-25 17:34:14,026::task::993::Storage.TaskManager.Task::(_decref) Task=`07cbef13-faf6-4f55-9130-e4fed9c4eacf`::ref 1 aborting False >Thread-70622::INFO::2014-09-25 17:34:14,030::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70622::DEBUG::2014-09-25 17:34:14,030::task::1191::Storage.TaskManager.Task::(prepare) Task=`07cbef13-faf6-4f55-9130-e4fed9c4eacf`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70622::DEBUG::2014-09-25 17:34:14,031::task::595::Storage.TaskManager.Task::(_updateState) Task=`07cbef13-faf6-4f55-9130-e4fed9c4eacf`::moving from state preparing -> state finished >Thread-70622::DEBUG::2014-09-25 17:34:14,031::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70622::DEBUG::2014-09-25 17:34:14,031::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70622::DEBUG::2014-09-25 17:34:14,032::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70622::DEBUG::2014-09-25 17:34:14,032::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70622::DEBUG::2014-09-25 17:34:14,032::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70622::DEBUG::2014-09-25 17:34:14,032::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70622::DEBUG::2014-09-25 17:34:14,033::task::993::Storage.TaskManager.Task::(_decref) Task=`07cbef13-faf6-4f55-9130-e4fed9c4eacf`::ref 0 aborting False >Thread-70622::DEBUG::2014-09-25 17:34:14,033::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70622::DEBUG::2014-09-25 17:34:14,034::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:14,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:14,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180419 s, 56.8 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 17:34:14,780::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:34:14,784::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:14,785::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:14,785::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:14,785::__init__::232::IOProcess::(_processLogs) (108646) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:34:14,786::__init__::232::IOProcess::(_processLogs) (108646) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:14,787::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:14,787::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:14,787::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:14,787::__init__::232::IOProcess::(_processLogs) (108647) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:34:14,788::__init__::232::IOProcess::(_processLogs) (108647) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:14,791::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:14,793::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:14,794::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:14,796::__init__::232::IOProcess::(_processLogs) (108648) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:34:14,798::__init__::232::IOProcess::(_processLogs) (108648) Queuing response >Thread-80::DEBUG::2014-09-25 17:34:14,801::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:34:14,802::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:14,802::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:14,803::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:14,803::__init__::232::IOProcess::(_processLogs) (108649) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:34:14,807::__init__::232::IOProcess::(_processLogs) (108649) Queuing response >Thread-80::DEBUG::2014-09-25 17:34:14,807::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:34:14,808::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:14,808::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:14,808::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:14,809::__init__::232::IOProcess::(_processLogs) (108650) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:34:14,809::__init__::232::IOProcess::(_processLogs) (108650) Queuing response >Thread-80::DEBUG::2014-09-25 17:34:14,809::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:34:14,811::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:14,812::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:14,812::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:14,813::__init__::232::IOProcess::(_processLogs) (108651) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:14,813::__init__::232::IOProcess::(_processLogs) (108651) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:14,813::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:14,814::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:14,814::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:14,814::__init__::232::IOProcess::(_processLogs) (108652) Got request for method 'glob' >Thread-80::DEBUG::2014-09-25 17:34:14,815::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:34:14,815::__init__::232::IOProcess::(_processLogs) (108652) Queuing response >Thread-80::INFO::2014-09-25 17:34:14,816::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:34:14,816::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:34:14,817::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:14,817::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:14,817::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:14,818::__init__::232::IOProcess::(_processLogs) (108653) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:14,818::__init__::232::IOProcess::(_processLogs) (108653) Queuing response >Thread-80::DEBUG::2014-09-25 17:34:14,819::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:34:14,830::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000320251 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:14,831::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:14,831::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:14,832::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:14,833::__init__::232::IOProcess::(_processLogs) (108654) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:14,833::__init__::232::IOProcess::(_processLogs) (108654) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:15,892::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:15,894::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70626::DEBUG::2014-09-25 17:34:15,897::task::595::Storage.TaskManager.Task::(_updateState) Task=`f2da7ce0-835f-46e3-8d12-31e1f7380cdb`::moving from state init -> state preparing >Thread-70626::INFO::2014-09-25 17:34:15,897::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70626::INFO::2014-09-25 17:34:15,898::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000489121', 'lastCheck': '9.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000320251', 'lastCheck': '1.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0278208', 'lastCheck': '4.6', 'valid': True}} >Thread-70626::DEBUG::2014-09-25 17:34:15,898::task::1191::Storage.TaskManager.Task::(prepare) Task=`f2da7ce0-835f-46e3-8d12-31e1f7380cdb`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000489121', 'lastCheck': '9.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000320251', 'lastCheck': '1.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0278208', 'lastCheck': '4.6', 'valid': True}} >Thread-70626::DEBUG::2014-09-25 17:34:15,898::task::595::Storage.TaskManager.Task::(_updateState) Task=`f2da7ce0-835f-46e3-8d12-31e1f7380cdb`::moving from state preparing -> state finished >Thread-70626::DEBUG::2014-09-25 17:34:15,899::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70626::DEBUG::2014-09-25 17:34:15,899::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70626::DEBUG::2014-09-25 17:34:15,899::task::993::Storage.TaskManager.Task::(_decref) Task=`f2da7ce0-835f-46e3-8d12-31e1f7380cdb`::ref 0 aborting False >Thread-70626::DEBUG::2014-09-25 17:34:15,901::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:15,926::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:15,927::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70627::DEBUG::2014-09-25 17:34:15,929::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:34:16,566::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:16,567::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:16,568::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:16,569::__init__::232::IOProcess::(_processLogs) (108655) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:16,571::__init__::232::IOProcess::(_processLogs) (108655) Queuing response >Thread-79::DEBUG::2014-09-25 17:34:16,573::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:34:16,589::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00132691 s, 424 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:16,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:16,593::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:16,595::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:16,596::__init__::232::IOProcess::(_processLogs) (108656) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:16,598::__init__::232::IOProcess::(_processLogs) (108656) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:16,600::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:16,602::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:16,603::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:16,605::__init__::232::IOProcess::(_processLogs) (108657) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:16,606::__init__::232::IOProcess::(_processLogs) (108657) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:16,608::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:16,609::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:16,610::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:16,613::__init__::232::IOProcess::(_processLogs) (108658) Got request for method 'access' >Dummy-148::DEBUG::2014-09-25 17:34:16,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:34:16,615::__init__::232::IOProcess::(_processLogs) (108658) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:16,626::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:16,628::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:16,629::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:16,630::__init__::232::IOProcess::(_processLogs) (108659) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:16,632::__init__::232::IOProcess::(_processLogs) (108659) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:16,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0352977 s, 29.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:34:18,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:18,990::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:18,992::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70628::DEBUG::2014-09-25 17:34:18,994::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:19,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021402 s, 47.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:34:21,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:21,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.01641 s, 62.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:21,339::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:21,339::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:21,339::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:21,340::__init__::232::IOProcess::(_processLogs) (108660) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:21,341::__init__::232::IOProcess::(_processLogs) (108660) Queuing response >Thread-81::DEBUG::2014-09-25 17:34:21,342::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:34:21,368::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0047373 s, 78.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:21,369::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:21,369::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:21,370::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:21,370::__init__::232::IOProcess::(_processLogs) (108661) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:21,373::__init__::232::IOProcess::(_processLogs) (108661) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:22,011::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:22,012::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70629::DEBUG::2014-09-25 17:34:22,014::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:23,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:23,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175235 s, 58.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:24,061::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70630::DEBUG::2014-09-25 17:34:24,062::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:34:24,062::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70630::DEBUG::2014-09-25 17:34:24,064::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a04295f-5384-4ce7-9201-fd7bdcc640f8`::moving from state init -> state preparing >Thread-70630::INFO::2014-09-25 17:34:24,065::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70630::INFO::2014-09-25 17:34:24,065::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70630::DEBUG::2014-09-25 17:34:24,066::task::1191::Storage.TaskManager.Task::(prepare) Task=`4a04295f-5384-4ce7-9201-fd7bdcc640f8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70630::DEBUG::2014-09-25 17:34:24,066::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a04295f-5384-4ce7-9201-fd7bdcc640f8`::moving from state preparing -> state finished >Thread-70630::DEBUG::2014-09-25 17:34:24,066::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70630::DEBUG::2014-09-25 17:34:24,066::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70630::DEBUG::2014-09-25 17:34:24,066::task::993::Storage.TaskManager.Task::(_decref) Task=`4a04295f-5384-4ce7-9201-fd7bdcc640f8`::ref 0 aborting False >Thread-70630::DEBUG::2014-09-25 17:34:24,067::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70630::DEBUG::2014-09-25 17:34:24,067::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:24,080::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:24,082::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70631::DEBUG::2014-09-25 17:34:24,082::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70631::DEBUG::2014-09-25 17:34:24,083::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d946f66-f461-4fb0-a75f-1c5289a17c67`::moving from state init -> state preparing >Thread-70631::INFO::2014-09-25 17:34:24,084::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70631::DEBUG::2014-09-25 17:34:24,084::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c8df26f9-3075-432a-9f8d-6afe964a6a56`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70631::DEBUG::2014-09-25 17:34:24,085::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70631::DEBUG::2014-09-25 17:34:24,085::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70631::DEBUG::2014-09-25 17:34:24,085::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c8df26f9-3075-432a-9f8d-6afe964a6a56`::Granted request >Thread-70631::DEBUG::2014-09-25 17:34:24,086::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`8d946f66-f461-4fb0-a75f-1c5289a17c67`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70631::DEBUG::2014-09-25 17:34:24,086::task::993::Storage.TaskManager.Task::(_decref) Task=`8d946f66-f461-4fb0-a75f-1c5289a17c67`::ref 1 aborting False >Thread-70631::INFO::2014-09-25 17:34:24,090::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70631::DEBUG::2014-09-25 17:34:24,091::task::1191::Storage.TaskManager.Task::(prepare) Task=`8d946f66-f461-4fb0-a75f-1c5289a17c67`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70631::DEBUG::2014-09-25 17:34:24,091::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d946f66-f461-4fb0-a75f-1c5289a17c67`::moving from state preparing -> state finished >Thread-70631::DEBUG::2014-09-25 17:34:24,091::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70631::DEBUG::2014-09-25 17:34:24,091::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70631::DEBUG::2014-09-25 17:34:24,092::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70631::DEBUG::2014-09-25 17:34:24,092::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70631::DEBUG::2014-09-25 17:34:24,092::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70631::DEBUG::2014-09-25 17:34:24,092::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70631::DEBUG::2014-09-25 17:34:24,093::task::993::Storage.TaskManager.Task::(_decref) Task=`8d946f66-f461-4fb0-a75f-1c5289a17c67`::ref 0 aborting False >Thread-70631::DEBUG::2014-09-25 17:34:24,093::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70631::DEBUG::2014-09-25 17:34:24,094::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:34:24,834::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:24,834::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:24,834::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:24,835::__init__::232::IOProcess::(_processLogs) (108662) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:24,835::__init__::232::IOProcess::(_processLogs) (108662) Queuing response >Thread-80::DEBUG::2014-09-25 17:34:24,836::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:34:24,845::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000480685 s, 786 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:24,846::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:24,846::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:24,846::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:24,846::__init__::232::IOProcess::(_processLogs) (108663) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:24,847::__init__::232::IOProcess::(_processLogs) (108663) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:25,030::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:25,032::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70632::DEBUG::2014-09-25 17:34:25,034::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:25,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:25,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.222089 s, 4.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:26,634::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:26,635::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:26,635::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:26,635::__init__::232::IOProcess::(_processLogs) (108664) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:26,638::__init__::232::IOProcess::(_processLogs) (108664) Queuing response >Thread-79::DEBUG::2014-09-25 17:34:26,639::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:34:26,659::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00804961 s, 69.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:26,660::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:26,661::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:26,661::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:26,661::__init__::232::IOProcess::(_processLogs) (108665) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:26,665::__init__::232::IOProcess::(_processLogs) (108665) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:26,667::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:26,667::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:26,667::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:26,668::__init__::232::IOProcess::(_processLogs) (108666) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:26,668::__init__::232::IOProcess::(_processLogs) (108666) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:26,669::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:26,669::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:26,669::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:26,670::__init__::232::IOProcess::(_processLogs) (108667) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:26,670::__init__::232::IOProcess::(_processLogs) (108667) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:26,671::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:26,671::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:26,672::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:26,673::__init__::232::IOProcess::(_processLogs) (108668) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:26,673::__init__::232::IOProcess::(_processLogs) (108668) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:27,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:27,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136695 s, 74.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:28,051::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:28,055::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70633::DEBUG::2014-09-25 17:34:28,060::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:29,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:29,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161143 s, 63.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:31,076::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:31,078::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70634::DEBUG::2014-09-25 17:34:31,081::task::595::Storage.TaskManager.Task::(_updateState) Task=`250ffa46-6ff5-447d-bd42-1aff6376d0e8`::moving from state init -> state preparing >Thread-70634::INFO::2014-09-25 17:34:31,081::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70634::INFO::2014-09-25 17:34:31,082::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00804961', 'lastCheck': '4.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000480685', 'lastCheck': '6.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0047373', 'lastCheck': '9.7', 'valid': True}} >Thread-70634::DEBUG::2014-09-25 17:34:31,082::task::1191::Storage.TaskManager.Task::(prepare) Task=`250ffa46-6ff5-447d-bd42-1aff6376d0e8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00804961', 'lastCheck': '4.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000480685', 'lastCheck': '6.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0047373', 'lastCheck': '9.7', 'valid': True}} >Thread-70634::DEBUG::2014-09-25 17:34:31,082::task::595::Storage.TaskManager.Task::(_updateState) Task=`250ffa46-6ff5-447d-bd42-1aff6376d0e8`::moving from state preparing -> state finished >Thread-70634::DEBUG::2014-09-25 17:34:31,082::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70634::DEBUG::2014-09-25 17:34:31,083::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70634::DEBUG::2014-09-25 17:34:31,083::task::993::Storage.TaskManager.Task::(_decref) Task=`250ffa46-6ff5-447d-bd42-1aff6376d0e8`::ref 0 aborting False >Thread-70634::DEBUG::2014-09-25 17:34:31,085::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:31,105::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:31,106::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70635::DEBUG::2014-09-25 17:34:31,108::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:34:31,375::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:31,375::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:31,376::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:31,376::__init__::232::IOProcess::(_processLogs) (108669) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:31,378::__init__::232::IOProcess::(_processLogs) (108669) Queuing response >Thread-81::DEBUG::2014-09-25 17:34:31,379::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:34:31,391::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00263549 s, 141 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:31,393::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:31,393::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:31,393::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:31,393::__init__::232::IOProcess::(_processLogs) (108670) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:31,396::__init__::232::IOProcess::(_processLogs) (108670) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:31,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:31,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0216389 s, 47.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:34:33,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:33,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147013 s, 69.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:34,125::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70636::DEBUG::2014-09-25 17:34:34,126::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:34:34,126::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70636::DEBUG::2014-09-25 17:34:34,129::task::595::Storage.TaskManager.Task::(_updateState) Task=`ce938991-8e2d-45bb-b01f-c89af656d61d`::moving from state init -> state preparing >Thread-70636::INFO::2014-09-25 17:34:34,129::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70636::INFO::2014-09-25 17:34:34,130::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70636::DEBUG::2014-09-25 17:34:34,130::task::1191::Storage.TaskManager.Task::(prepare) Task=`ce938991-8e2d-45bb-b01f-c89af656d61d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70636::DEBUG::2014-09-25 17:34:34,130::task::595::Storage.TaskManager.Task::(_updateState) Task=`ce938991-8e2d-45bb-b01f-c89af656d61d`::moving from state preparing -> state finished >Thread-70636::DEBUG::2014-09-25 17:34:34,130::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70636::DEBUG::2014-09-25 17:34:34,131::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70636::DEBUG::2014-09-25 17:34:34,131::task::993::Storage.TaskManager.Task::(_decref) Task=`ce938991-8e2d-45bb-b01f-c89af656d61d`::ref 0 aborting False >Thread-70636::DEBUG::2014-09-25 17:34:34,131::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70636::DEBUG::2014-09-25 17:34:34,132::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:34,145::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:34,146::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70637::DEBUG::2014-09-25 17:34:34,146::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70637::DEBUG::2014-09-25 17:34:34,148::task::595::Storage.TaskManager.Task::(_updateState) Task=`a758c69b-24c7-4697-bebe-26491a1f3e10`::moving from state init -> state preparing >Thread-70637::INFO::2014-09-25 17:34:34,148::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70637::DEBUG::2014-09-25 17:34:34,149::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ff5f44fd-5226-48f2-a639-ceb915fc5bdd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70637::DEBUG::2014-09-25 17:34:34,149::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70637::DEBUG::2014-09-25 17:34:34,149::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70637::DEBUG::2014-09-25 17:34:34,149::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ff5f44fd-5226-48f2-a639-ceb915fc5bdd`::Granted request >Thread-70637::DEBUG::2014-09-25 17:34:34,150::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a758c69b-24c7-4697-bebe-26491a1f3e10`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70637::DEBUG::2014-09-25 17:34:34,150::task::993::Storage.TaskManager.Task::(_decref) Task=`a758c69b-24c7-4697-bebe-26491a1f3e10`::ref 1 aborting False >Thread-70637::INFO::2014-09-25 17:34:34,155::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70637::DEBUG::2014-09-25 17:34:34,155::task::1191::Storage.TaskManager.Task::(prepare) Task=`a758c69b-24c7-4697-bebe-26491a1f3e10`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70637::DEBUG::2014-09-25 17:34:34,156::task::595::Storage.TaskManager.Task::(_updateState) Task=`a758c69b-24c7-4697-bebe-26491a1f3e10`::moving from state preparing -> state finished >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:34,156::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70637::DEBUG::2014-09-25 17:34:34,157::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70637::DEBUG::2014-09-25 17:34:34,158::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >JsonRpcServer::DEBUG::2014-09-25 17:34:34,159::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70637::DEBUG::2014-09-25 17:34:34,161::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70637::DEBUG::2014-09-25 17:34:34,163::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70637::DEBUG::2014-09-25 17:34:34,164::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70637::DEBUG::2014-09-25 17:34:34,164::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70638::DEBUG::2014-09-25 17:34:34,165::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-70637::DEBUG::2014-09-25 17:34:34,167::task::993::Storage.TaskManager.Task::(_decref) Task=`a758c69b-24c7-4697-bebe-26491a1f3e10`::ref 0 aborting False >Thread-70637::DEBUG::2014-09-25 17:34:34,169::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70637::DEBUG::2014-09-25 17:34:34,170::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:34:34,849::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:34,850::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:34,850::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:34,850::__init__::232::IOProcess::(_processLogs) (108671) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:34,850::__init__::232::IOProcess::(_processLogs) (108671) Queuing response >Thread-80::DEBUG::2014-09-25 17:34:34,852::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:34:34,861::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000475831 s, 794 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:34,862::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:34,862::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:34,862::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:34,863::__init__::232::IOProcess::(_processLogs) (108672) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:34,863::__init__::232::IOProcess::(_processLogs) (108672) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:35,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:35,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133027 s, 77.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:36,673::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:36,674::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:36,674::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:36,674::__init__::232::IOProcess::(_processLogs) (108673) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:36,676::__init__::232::IOProcess::(_processLogs) (108673) Queuing response >Thread-79::DEBUG::2014-09-25 17:34:36,676::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:34:36,692::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00489566 s, 115 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:36,694::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:36,694::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:36,694::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:36,695::__init__::232::IOProcess::(_processLogs) (108674) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:36,698::__init__::232::IOProcess::(_processLogs) (108674) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:36,699::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:36,699::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:36,700::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:36,700::__init__::232::IOProcess::(_processLogs) (108675) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:36,700::__init__::232::IOProcess::(_processLogs) (108675) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:36,701::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:36,701::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:36,701::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:36,702::__init__::232::IOProcess::(_processLogs) (108676) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:36,702::__init__::232::IOProcess::(_processLogs) (108676) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:36,703::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:36,703::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:36,704::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:36,705::__init__::232::IOProcess::(_processLogs) (108677) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:36,705::__init__::232::IOProcess::(_processLogs) (108677) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:37,194::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:37,195::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70639::DEBUG::2014-09-25 17:34:37,197::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:37,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:37,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156692 s, 65.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:34:39,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:39,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0474564 s, 21.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:40,215::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:40,216::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70640::DEBUG::2014-09-25 17:34:40,218::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:34:41,398::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:41,398::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:41,399::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:41,399::__init__::232::IOProcess::(_processLogs) (108678) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:41,401::__init__::232::IOProcess::(_processLogs) (108678) Queuing response >Thread-81::DEBUG::2014-09-25 17:34:41,402::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:34:41,417::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00510427 s, 72.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:41,419::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:41,419::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:41,419::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:41,419::__init__::232::IOProcess::(_processLogs) (108679) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:41,420::__init__::232::IOProcess::(_processLogs) (108679) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:41,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:41,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171078 s, 59.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:43,230::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:43,231::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70641::DEBUG::2014-09-25 17:34:43,233::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:43,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:43,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155025 s, 66.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:44,201::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:44,202::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70642::DEBUG::2014-09-25 17:34:44,202::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70642::DEBUG::2014-09-25 17:34:44,204::task::595::Storage.TaskManager.Task::(_updateState) Task=`e9865d6b-5d34-445b-8fbb-7544090e9704`::moving from state init -> state preparing >Thread-70642::INFO::2014-09-25 17:34:44,205::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70642::INFO::2014-09-25 17:34:44,205::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70642::DEBUG::2014-09-25 17:34:44,206::task::1191::Storage.TaskManager.Task::(prepare) Task=`e9865d6b-5d34-445b-8fbb-7544090e9704`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70642::DEBUG::2014-09-25 17:34:44,206::task::595::Storage.TaskManager.Task::(_updateState) Task=`e9865d6b-5d34-445b-8fbb-7544090e9704`::moving from state preparing -> state finished >Thread-70642::DEBUG::2014-09-25 17:34:44,206::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70642::DEBUG::2014-09-25 17:34:44,206::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70642::DEBUG::2014-09-25 17:34:44,207::task::993::Storage.TaskManager.Task::(_decref) Task=`e9865d6b-5d34-445b-8fbb-7544090e9704`::ref 0 aborting False >Thread-70642::DEBUG::2014-09-25 17:34:44,207::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70642::DEBUG::2014-09-25 17:34:44,207::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:44,222::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:44,223::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70643::DEBUG::2014-09-25 17:34:44,223::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70643::DEBUG::2014-09-25 17:34:44,225::task::595::Storage.TaskManager.Task::(_updateState) Task=`c152ea7c-be58-4740-89e1-097605852633`::moving from state init -> state preparing >Thread-70643::INFO::2014-09-25 17:34:44,225::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70643::DEBUG::2014-09-25 17:34:44,226::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`004f2438-b011-4432-b303-421e31e25921`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70643::DEBUG::2014-09-25 17:34:44,226::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70643::DEBUG::2014-09-25 17:34:44,227::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70643::DEBUG::2014-09-25 17:34:44,227::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`004f2438-b011-4432-b303-421e31e25921`::Granted request >Thread-70643::DEBUG::2014-09-25 17:34:44,228::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c152ea7c-be58-4740-89e1-097605852633`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70643::DEBUG::2014-09-25 17:34:44,228::task::993::Storage.TaskManager.Task::(_decref) Task=`c152ea7c-be58-4740-89e1-097605852633`::ref 1 aborting False >Thread-70643::INFO::2014-09-25 17:34:44,232::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70643::DEBUG::2014-09-25 17:34:44,233::task::1191::Storage.TaskManager.Task::(prepare) Task=`c152ea7c-be58-4740-89e1-097605852633`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70643::DEBUG::2014-09-25 17:34:44,233::task::595::Storage.TaskManager.Task::(_updateState) Task=`c152ea7c-be58-4740-89e1-097605852633`::moving from state preparing -> state finished >Thread-70643::DEBUG::2014-09-25 17:34:44,233::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70643::DEBUG::2014-09-25 17:34:44,233::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70643::DEBUG::2014-09-25 17:34:44,234::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70643::DEBUG::2014-09-25 17:34:44,234::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70643::DEBUG::2014-09-25 17:34:44,234::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70643::DEBUG::2014-09-25 17:34:44,235::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70643::DEBUG::2014-09-25 17:34:44,235::task::993::Storage.TaskManager.Task::(_decref) Task=`c152ea7c-be58-4740-89e1-097605852633`::ref 0 aborting False >Thread-70643::DEBUG::2014-09-25 17:34:44,235::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70643::DEBUG::2014-09-25 17:34:44,236::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:34:44,865::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:44,866::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:44,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:44,866::__init__::232::IOProcess::(_processLogs) (108680) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:44,867::__init__::232::IOProcess::(_processLogs) (108680) Queuing response >Thread-80::DEBUG::2014-09-25 17:34:44,868::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:34:44,878::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000510645 s, 740 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:44,879::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:44,879::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:44,880::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:44,880::__init__::232::IOProcess::(_processLogs) (108681) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:44,880::__init__::232::IOProcess::(_processLogs) (108681) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:45,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:45,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136894 s, 74.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:46,248::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:46,252::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70644::DEBUG::2014-09-25 17:34:46,261::task::595::Storage.TaskManager.Task::(_updateState) Task=`a80311ce-ce4c-4180-8d8b-fb53b6c5a887`::moving from state init -> state preparing >Thread-70644::INFO::2014-09-25 17:34:46,262::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70644::INFO::2014-09-25 17:34:46,264::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00489566', 'lastCheck': '9.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000510645', 'lastCheck': '1.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00510427', 'lastCheck': '4.8', 'valid': True}} >Thread-70644::DEBUG::2014-09-25 17:34:46,265::task::1191::Storage.TaskManager.Task::(prepare) Task=`a80311ce-ce4c-4180-8d8b-fb53b6c5a887`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00489566', 'lastCheck': '9.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000510645', 'lastCheck': '1.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00510427', 'lastCheck': '4.8', 'valid': True}} >Thread-70644::DEBUG::2014-09-25 17:34:46,267::task::595::Storage.TaskManager.Task::(_updateState) Task=`a80311ce-ce4c-4180-8d8b-fb53b6c5a887`::moving from state preparing -> state finished >Thread-70644::DEBUG::2014-09-25 17:34:46,268::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70644::DEBUG::2014-09-25 17:34:46,269::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70644::DEBUG::2014-09-25 17:34:46,270::task::993::Storage.TaskManager.Task::(_decref) Task=`a80311ce-ce4c-4180-8d8b-fb53b6c5a887`::ref 0 aborting False >Thread-70644::DEBUG::2014-09-25 17:34:46,279::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:46,302::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:46,305::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70645::DEBUG::2014-09-25 17:34:46,311::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:34:46,707::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:46,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:46,709::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:46,710::__init__::232::IOProcess::(_processLogs) (108682) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:46,712::__init__::232::IOProcess::(_processLogs) (108682) Queuing response >Thread-79::DEBUG::2014-09-25 17:34:46,714::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:34:46,725::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000514049 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:46,728::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:46,729::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:46,730::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:46,731::__init__::232::IOProcess::(_processLogs) (108683) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:46,733::__init__::232::IOProcess::(_processLogs) (108683) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:46,736::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:46,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:46,738::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:46,740::__init__::232::IOProcess::(_processLogs) (108684) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:46,742::__init__::232::IOProcess::(_processLogs) (108684) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:46,744::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:46,745::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:46,746::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:46,748::__init__::232::IOProcess::(_processLogs) (108685) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:46,749::__init__::232::IOProcess::(_processLogs) (108685) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:46,752::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:46,753::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:46,754::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:46,755::__init__::232::IOProcess::(_processLogs) (108686) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:46,757::__init__::232::IOProcess::(_processLogs) (108686) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:47,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:48,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168426 s, 60.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:49,366::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:49,368::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70646::DEBUG::2014-09-25 17:34:49,374::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:50,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:50,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136987 s, 74.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:51,422::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:51,423::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:51,423::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:51,423::__init__::232::IOProcess::(_processLogs) (108687) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:51,425::__init__::232::IOProcess::(_processLogs) (108687) Queuing response >Thread-81::DEBUG::2014-09-25 17:34:51,426::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:34:51,443::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00359589 s, 103 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:51,445::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:51,445::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:51,445::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:51,445::__init__::232::IOProcess::(_processLogs) (108688) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:51,447::__init__::232::IOProcess::(_processLogs) (108688) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:52,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:52,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013689 s, 74.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:52,402::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:52,404::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70647::DEBUG::2014-09-25 17:34:52,406::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:54,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:54,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184096 s, 55.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:54,268::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:54,269::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70648::DEBUG::2014-09-25 17:34:54,270::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70648::DEBUG::2014-09-25 17:34:54,272::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f50018e-35cc-4df0-a0ae-bea3593b7b66`::moving from state init -> state preparing >Thread-70648::INFO::2014-09-25 17:34:54,272::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70648::INFO::2014-09-25 17:34:54,273::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70648::DEBUG::2014-09-25 17:34:54,273::task::1191::Storage.TaskManager.Task::(prepare) Task=`9f50018e-35cc-4df0-a0ae-bea3593b7b66`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70648::DEBUG::2014-09-25 17:34:54,273::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f50018e-35cc-4df0-a0ae-bea3593b7b66`::moving from state preparing -> state finished >Thread-70648::DEBUG::2014-09-25 17:34:54,273::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70648::DEBUG::2014-09-25 17:34:54,273::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70648::DEBUG::2014-09-25 17:34:54,274::task::993::Storage.TaskManager.Task::(_decref) Task=`9f50018e-35cc-4df0-a0ae-bea3593b7b66`::ref 0 aborting False >Thread-70648::DEBUG::2014-09-25 17:34:54,274::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70648::DEBUG::2014-09-25 17:34:54,274::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:54,288::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:54,289::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70649::DEBUG::2014-09-25 17:34:54,289::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70649::DEBUG::2014-09-25 17:34:54,291::task::595::Storage.TaskManager.Task::(_updateState) Task=`26904e02-5da9-4752-a837-3b21069f6941`::moving from state init -> state preparing >Thread-70649::INFO::2014-09-25 17:34:54,291::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70649::DEBUG::2014-09-25 17:34:54,292::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`924a7167-a91c-43b7-bacc-6dad4d1472cc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70649::DEBUG::2014-09-25 17:34:54,292::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70649::DEBUG::2014-09-25 17:34:54,293::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70649::DEBUG::2014-09-25 17:34:54,293::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`924a7167-a91c-43b7-bacc-6dad4d1472cc`::Granted request >Thread-70649::DEBUG::2014-09-25 17:34:54,293::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`26904e02-5da9-4752-a837-3b21069f6941`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70649::DEBUG::2014-09-25 17:34:54,294::task::993::Storage.TaskManager.Task::(_decref) Task=`26904e02-5da9-4752-a837-3b21069f6941`::ref 1 aborting False >Thread-70649::INFO::2014-09-25 17:34:54,298::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70649::DEBUG::2014-09-25 17:34:54,298::task::1191::Storage.TaskManager.Task::(prepare) Task=`26904e02-5da9-4752-a837-3b21069f6941`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70649::DEBUG::2014-09-25 17:34:54,299::task::595::Storage.TaskManager.Task::(_updateState) Task=`26904e02-5da9-4752-a837-3b21069f6941`::moving from state preparing -> state finished >Thread-70649::DEBUG::2014-09-25 17:34:54,299::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70649::DEBUG::2014-09-25 17:34:54,299::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70649::DEBUG::2014-09-25 17:34:54,299::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70649::DEBUG::2014-09-25 17:34:54,300::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70649::DEBUG::2014-09-25 17:34:54,300::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70649::DEBUG::2014-09-25 17:34:54,300::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70649::DEBUG::2014-09-25 17:34:54,301::task::993::Storage.TaskManager.Task::(_decref) Task=`26904e02-5da9-4752-a837-3b21069f6941`::ref 0 aborting False >Thread-70649::DEBUG::2014-09-25 17:34:54,301::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70649::DEBUG::2014-09-25 17:34:54,302::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:34:54,883::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:54,883::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:54,883::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:34:54,885::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:34:54,885::__init__::232::IOProcess::(_processLogs) (108689) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:54,894::__init__::232::IOProcess::(_processLogs) (108689) Queuing response >Thread-80::DEBUG::2014-09-25 17:34:54,895::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000496805 s, 761 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:54,896::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:54,896::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:54,896::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:54,896::__init__::232::IOProcess::(_processLogs) (108690) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:54,897::__init__::232::IOProcess::(_processLogs) (108690) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:55,420::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:55,421::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70650::DEBUG::2014-09-25 17:34:55,423::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:34:56,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:56,201::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142656 s, 71.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:56,759::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:56,759::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:56,760::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:56,760::__init__::232::IOProcess::(_processLogs) (108691) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:56,761::__init__::232::IOProcess::(_processLogs) (108691) Queuing response >Thread-79::DEBUG::2014-09-25 17:34:56,762::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:34:56,777::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00465022 s, 121 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:34:56,778::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:56,778::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:56,778::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:56,779::__init__::232::IOProcess::(_processLogs) (108692) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:34:56,785::__init__::232::IOProcess::(_processLogs) (108692) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:56,786::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:56,787::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:56,787::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:56,788::__init__::232::IOProcess::(_processLogs) (108693) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:56,788::__init__::232::IOProcess::(_processLogs) (108693) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:56,788::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:56,789::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:56,789::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:56,790::__init__::232::IOProcess::(_processLogs) (108694) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:56,790::__init__::232::IOProcess::(_processLogs) (108694) Queuing response >Thread-29::DEBUG::2014-09-25 17:34:56,790::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:34:56,791::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:34:56,791::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:34:56,792::__init__::232::IOProcess::(_processLogs) (108695) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:34:56,792::__init__::232::IOProcess::(_processLogs) (108695) Queuing response >Dummy-148::DEBUG::2014-09-25 17:34:58,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:34:58,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0312188 s, 32.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:34:58,437::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:34:58,438::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70651::DEBUG::2014-09-25 17:34:58,440::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:00,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:00,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133462 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:01,449::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:01,449::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:01,450::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:01,450::__init__::232::IOProcess::(_processLogs) (108696) Got request for method 'statvfs' >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:01,456::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:01,457::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70652::DEBUG::2014-09-25 17:35:01,459::task::595::Storage.TaskManager.Task::(_updateState) Task=`39d6263e-0083-445c-bd5b-1b1a8f9a89c4`::moving from state init -> state preparing >Thread-70652::INFO::2014-09-25 17:35:01,460::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70652::INFO::2014-09-25 17:35:01,460::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00465022', 'lastCheck': '4.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000496805', 'lastCheck': '6.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00359589', 'lastCheck': '10.0', 'valid': True}} >Thread-70652::DEBUG::2014-09-25 17:35:01,460::task::1191::Storage.TaskManager.Task::(prepare) Task=`39d6263e-0083-445c-bd5b-1b1a8f9a89c4`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00465022', 'lastCheck': '4.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000496805', 'lastCheck': '6.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00359589', 'lastCheck': '10.0', 'valid': True}} >Thread-70652::DEBUG::2014-09-25 17:35:01,461::task::595::Storage.TaskManager.Task::(_updateState) Task=`39d6263e-0083-445c-bd5b-1b1a8f9a89c4`::moving from state preparing -> state finished >Thread-70652::DEBUG::2014-09-25 17:35:01,461::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70652::DEBUG::2014-09-25 17:35:01,461::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70652::DEBUG::2014-09-25 17:35:01,461::task::993::Storage.TaskManager.Task::(_decref) Task=`39d6263e-0083-445c-bd5b-1b1a8f9a89c4`::ref 0 aborting False >Thread-29::DEBUG::2014-09-25 17:35:01,462::__init__::232::IOProcess::(_processLogs) (108696) Queuing response >Thread-81::DEBUG::2014-09-25 17:35:01,465::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-70652::DEBUG::2014-09-25 17:35:01,475::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:01,497::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-81::DEBUG::2014-09-25 17:35:01,497::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00748816 s, 49.5 kB/s\n'; <rc> = 0 >JsonRpcServer::DEBUG::2014-09-25 17:35:01,501::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-29::DEBUG::2014-09-25 17:35:01,502::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:01,503::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:01,504::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:01,506::__init__::232::IOProcess::(_processLogs) (108697) Got request for method 'statvfs' >Thread-70653::DEBUG::2014-09-25 17:35:01,507::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:35:01,509::__init__::232::IOProcess::(_processLogs) (108697) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:02,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:02,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0197059 s, 52.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:04,335::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70654::DEBUG::2014-09-25 17:35:04,336::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:35:04,336::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70654::DEBUG::2014-09-25 17:35:04,339::task::595::Storage.TaskManager.Task::(_updateState) Task=`c04353e4-8be7-4c07-a471-680e5a2d3ac0`::moving from state init -> state preparing >Thread-70654::INFO::2014-09-25 17:35:04,339::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70654::INFO::2014-09-25 17:35:04,339::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70654::DEBUG::2014-09-25 17:35:04,340::task::1191::Storage.TaskManager.Task::(prepare) Task=`c04353e4-8be7-4c07-a471-680e5a2d3ac0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70654::DEBUG::2014-09-25 17:35:04,340::task::595::Storage.TaskManager.Task::(_updateState) Task=`c04353e4-8be7-4c07-a471-680e5a2d3ac0`::moving from state preparing -> state finished >Thread-70654::DEBUG::2014-09-25 17:35:04,340::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70654::DEBUG::2014-09-25 17:35:04,340::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70654::DEBUG::2014-09-25 17:35:04,341::task::993::Storage.TaskManager.Task::(_decref) Task=`c04353e4-8be7-4c07-a471-680e5a2d3ac0`::ref 0 aborting False >Thread-70654::DEBUG::2014-09-25 17:35:04,341::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70654::DEBUG::2014-09-25 17:35:04,341::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:04,374::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:04,374::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70655::DEBUG::2014-09-25 17:35:04,375::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70655::DEBUG::2014-09-25 17:35:04,376::task::595::Storage.TaskManager.Task::(_updateState) Task=`4c5cdfa1-629c-47a7-aa54-e978f23031da`::moving from state init -> state preparing >Thread-70655::INFO::2014-09-25 17:35:04,376::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70655::DEBUG::2014-09-25 17:35:04,377::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7ecdbc40-19d6-40d7-8e66-66010ba6774a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70655::DEBUG::2014-09-25 17:35:04,377::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70655::DEBUG::2014-09-25 17:35:04,378::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70655::DEBUG::2014-09-25 17:35:04,378::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7ecdbc40-19d6-40d7-8e66-66010ba6774a`::Granted request >Thread-70655::DEBUG::2014-09-25 17:35:04,379::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4c5cdfa1-629c-47a7-aa54-e978f23031da`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70655::DEBUG::2014-09-25 17:35:04,379::task::993::Storage.TaskManager.Task::(_decref) Task=`4c5cdfa1-629c-47a7-aa54-e978f23031da`::ref 1 aborting False >Thread-70655::INFO::2014-09-25 17:35:04,383::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70655::DEBUG::2014-09-25 17:35:04,383::task::1191::Storage.TaskManager.Task::(prepare) Task=`4c5cdfa1-629c-47a7-aa54-e978f23031da`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70655::DEBUG::2014-09-25 17:35:04,384::task::595::Storage.TaskManager.Task::(_updateState) Task=`4c5cdfa1-629c-47a7-aa54-e978f23031da`::moving from state preparing -> state finished >Thread-70655::DEBUG::2014-09-25 17:35:04,384::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70655::DEBUG::2014-09-25 17:35:04,384::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70655::DEBUG::2014-09-25 17:35:04,384::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70655::DEBUG::2014-09-25 17:35:04,385::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70655::DEBUG::2014-09-25 17:35:04,385::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70655::DEBUG::2014-09-25 17:35:04,385::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70655::DEBUG::2014-09-25 17:35:04,386::task::993::Storage.TaskManager.Task::(_decref) Task=`4c5cdfa1-629c-47a7-aa54-e978f23031da`::ref 0 aborting False >Thread-70655::DEBUG::2014-09-25 17:35:04,386::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70655::DEBUG::2014-09-25 17:35:04,386::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:04,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:04,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174912 s, 58.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:04,566::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:04,567::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70656::DEBUG::2014-09-25 17:35:04,569::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:35:04,899::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:04,899::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:04,900::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:04,900::__init__::232::IOProcess::(_processLogs) (108698) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:04,900::__init__::232::IOProcess::(_processLogs) (108698) Queuing response >Thread-80::DEBUG::2014-09-25 17:35:04,901::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:35:04,910::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000392683 s, 963 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:04,911::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:04,912::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:04,912::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:04,912::__init__::232::IOProcess::(_processLogs) (108699) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:04,913::__init__::232::IOProcess::(_processLogs) (108699) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:06,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:06,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166551 s, 61.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:06,793::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:06,793::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:06,794::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:06,794::__init__::232::IOProcess::(_processLogs) (108700) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:06,794::__init__::232::IOProcess::(_processLogs) (108700) Queuing response >Thread-79::DEBUG::2014-09-25 17:35:06,796::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:35:06,804::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000461575 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:06,806::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:06,806::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:06,806::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:06,806::__init__::232::IOProcess::(_processLogs) (108701) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:06,807::__init__::232::IOProcess::(_processLogs) (108701) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:06,808::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:06,809::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:06,809::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:06,810::__init__::232::IOProcess::(_processLogs) (108702) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:06,810::__init__::232::IOProcess::(_processLogs) (108702) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:06,810::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:06,811::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:06,812::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:06,812::__init__::232::IOProcess::(_processLogs) (108703) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:06,812::__init__::232::IOProcess::(_processLogs) (108703) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:06,812::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:06,813::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:06,814::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:06,814::__init__::232::IOProcess::(_processLogs) (108704) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:06,814::__init__::232::IOProcess::(_processLogs) (108704) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:07,584::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:07,585::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70657::DEBUG::2014-09-25 17:35:07,587::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:08,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:08,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166489 s, 61.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:10,603::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:10,605::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70658::DEBUG::2014-09-25 17:35:10,607::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:10,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:10,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129866 s, 78.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:11,511::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:11,512::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:11,512::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:11,512::__init__::232::IOProcess::(_processLogs) (108705) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:11,516::__init__::232::IOProcess::(_processLogs) (108705) Queuing response >Thread-81::DEBUG::2014-09-25 17:35:11,517::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:35:11,541::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00843457 s, 44.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:11,542::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:11,543::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:11,543::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:11,543::__init__::232::IOProcess::(_processLogs) (108706) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:11,548::__init__::232::IOProcess::(_processLogs) (108706) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:12,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:12,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192773 s, 53.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:13,625::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:13,626::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70659::DEBUG::2014-09-25 17:35:13,628::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:14,418::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70660::DEBUG::2014-09-25 17:35:14,420::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:35:14,420::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70660::DEBUG::2014-09-25 17:35:14,422::task::595::Storage.TaskManager.Task::(_updateState) Task=`6477f873-893c-4101-9416-d0e03722ad09`::moving from state init -> state preparing >Thread-70660::INFO::2014-09-25 17:35:14,422::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70660::INFO::2014-09-25 17:35:14,423::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70660::DEBUG::2014-09-25 17:35:14,423::task::1191::Storage.TaskManager.Task::(prepare) Task=`6477f873-893c-4101-9416-d0e03722ad09`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70660::DEBUG::2014-09-25 17:35:14,423::task::595::Storage.TaskManager.Task::(_updateState) Task=`6477f873-893c-4101-9416-d0e03722ad09`::moving from state preparing -> state finished >Thread-70660::DEBUG::2014-09-25 17:35:14,424::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70660::DEBUG::2014-09-25 17:35:14,424::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70660::DEBUG::2014-09-25 17:35:14,424::task::993::Storage.TaskManager.Task::(_decref) Task=`6477f873-893c-4101-9416-d0e03722ad09`::ref 0 aborting False >Thread-70660::DEBUG::2014-09-25 17:35:14,424::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70660::DEBUG::2014-09-25 17:35:14,425::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:14,438::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70661::DEBUG::2014-09-25 17:35:14,439::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:35:14,439::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70661::DEBUG::2014-09-25 17:35:14,441::task::595::Storage.TaskManager.Task::(_updateState) Task=`41341a72-34e1-4981-8615-bc159e71bfce`::moving from state init -> state preparing >Thread-70661::INFO::2014-09-25 17:35:14,442::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70661::DEBUG::2014-09-25 17:35:14,442::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`667223c6-7672-4bea-82cc-0b78b662db0b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70661::DEBUG::2014-09-25 17:35:14,443::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70661::DEBUG::2014-09-25 17:35:14,443::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70661::DEBUG::2014-09-25 17:35:14,443::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`667223c6-7672-4bea-82cc-0b78b662db0b`::Granted request >Thread-70661::DEBUG::2014-09-25 17:35:14,444::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`41341a72-34e1-4981-8615-bc159e71bfce`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70661::DEBUG::2014-09-25 17:35:14,444::task::993::Storage.TaskManager.Task::(_decref) Task=`41341a72-34e1-4981-8615-bc159e71bfce`::ref 1 aborting False >Thread-70661::INFO::2014-09-25 17:35:14,449::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70661::DEBUG::2014-09-25 17:35:14,449::task::1191::Storage.TaskManager.Task::(prepare) Task=`41341a72-34e1-4981-8615-bc159e71bfce`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70661::DEBUG::2014-09-25 17:35:14,450::task::595::Storage.TaskManager.Task::(_updateState) Task=`41341a72-34e1-4981-8615-bc159e71bfce`::moving from state preparing -> state finished >Thread-70661::DEBUG::2014-09-25 17:35:14,450::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70661::DEBUG::2014-09-25 17:35:14,450::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70661::DEBUG::2014-09-25 17:35:14,450::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70661::DEBUG::2014-09-25 17:35:14,451::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70661::DEBUG::2014-09-25 17:35:14,451::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70661::DEBUG::2014-09-25 17:35:14,451::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70661::DEBUG::2014-09-25 17:35:14,452::task::993::Storage.TaskManager.Task::(_decref) Task=`41341a72-34e1-4981-8615-bc159e71bfce`::ref 0 aborting False >Thread-70661::DEBUG::2014-09-25 17:35:14,452::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70661::DEBUG::2014-09-25 17:35:14,453::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:14,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:14,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0219238 s, 46.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:14,915::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:14,915::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:14,915::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:14,916::__init__::232::IOProcess::(_processLogs) (108707) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:14,916::__init__::232::IOProcess::(_processLogs) (108707) Queuing response >Thread-80::DEBUG::2014-09-25 17:35:14,918::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:35:14,926::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000513001 s, 737 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:14,927::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:14,928::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:14,928::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:14,928::__init__::232::IOProcess::(_processLogs) (108708) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:14,929::__init__::232::IOProcess::(_processLogs) (108708) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:16,650::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:16,653::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70662::DEBUG::2014-09-25 17:35:16,662::task::595::Storage.TaskManager.Task::(_updateState) Task=`28459318-950a-431a-b974-5d602564cb80`::moving from state init -> state preparing >Thread-70662::INFO::2014-09-25 17:35:16,663::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70662::INFO::2014-09-25 17:35:16,665::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000461575', 'lastCheck': '9.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000513001', 'lastCheck': '1.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00843457', 'lastCheck': '5.1', 'valid': True}} >Thread-70662::DEBUG::2014-09-25 17:35:16,666::task::1191::Storage.TaskManager.Task::(prepare) Task=`28459318-950a-431a-b974-5d602564cb80`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000461575', 'lastCheck': '9.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000513001', 'lastCheck': '1.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00843457', 'lastCheck': '5.1', 'valid': True}} >Thread-70662::DEBUG::2014-09-25 17:35:16,667::task::595::Storage.TaskManager.Task::(_updateState) Task=`28459318-950a-431a-b974-5d602564cb80`::moving from state preparing -> state finished >Thread-70662::DEBUG::2014-09-25 17:35:16,668::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70662::DEBUG::2014-09-25 17:35:16,669::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70662::DEBUG::2014-09-25 17:35:16,670::task::993::Storage.TaskManager.Task::(_decref) Task=`28459318-950a-431a-b974-5d602564cb80`::ref 0 aborting False >Thread-70662::DEBUG::2014-09-25 17:35:16,679::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:16,709::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:16,712::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70663::DEBUG::2014-09-25 17:35:16,717::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:16,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:35:16,816::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:16,819::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:16,821::__init__::232::IOProcess::(_processLogs) Extracting request information... >Dummy-148::DEBUG::2014-09-25 17:35:16,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0254714 s, 40.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:16,824::__init__::232::IOProcess::(_processLogs) (108709) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:16,828::__init__::232::IOProcess::(_processLogs) (108709) Queuing response >Thread-79::DEBUG::2014-09-25 17:35:16,830::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:35:16,854::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0005639 s, 997 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:16,858::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:16,860::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:16,861::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:16,863::__init__::232::IOProcess::(_processLogs) (108710) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:16,866::__init__::232::IOProcess::(_processLogs) (108710) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:16,869::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:16,870::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:16,872::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:16,874::__init__::232::IOProcess::(_processLogs) (108711) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:16,876::__init__::232::IOProcess::(_processLogs) (108711) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:16,880::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:16,881::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:16,883::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:16,885::__init__::232::IOProcess::(_processLogs) (108712) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:16,886::__init__::232::IOProcess::(_processLogs) (108712) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:16,890::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:16,892::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:16,893::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:16,895::__init__::232::IOProcess::(_processLogs) (108713) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:16,897::__init__::232::IOProcess::(_processLogs) (108713) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:19,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:19,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163223 s, 62.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:19,774::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:19,775::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70664::DEBUG::2014-09-25 17:35:19,778::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:21,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:21,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134218 s, 76.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:21,550::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:21,550::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:21,550::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:21,551::__init__::232::IOProcess::(_processLogs) (108714) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:21,557::__init__::232::IOProcess::(_processLogs) (108714) Queuing response >Thread-81::DEBUG::2014-09-25 17:35:21,558::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:35:21,578::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00508118 s, 73.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:21,579::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:21,579::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:21,579::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:21,580::__init__::232::IOProcess::(_processLogs) (108715) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:21,582::__init__::232::IOProcess::(_processLogs) (108715) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:22,793::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:22,795::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70665::DEBUG::2014-09-25 17:35:22,797::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:23,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:23,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160561 s, 63.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:24,485::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70666::DEBUG::2014-09-25 17:35:24,486::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:35:24,486::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70666::DEBUG::2014-09-25 17:35:24,489::task::595::Storage.TaskManager.Task::(_updateState) Task=`aa25b968-9153-4060-ab43-12c24f9ec2f4`::moving from state init -> state preparing >Thread-70666::INFO::2014-09-25 17:35:24,489::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70666::INFO::2014-09-25 17:35:24,490::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70666::DEBUG::2014-09-25 17:35:24,490::task::1191::Storage.TaskManager.Task::(prepare) Task=`aa25b968-9153-4060-ab43-12c24f9ec2f4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70666::DEBUG::2014-09-25 17:35:24,490::task::595::Storage.TaskManager.Task::(_updateState) Task=`aa25b968-9153-4060-ab43-12c24f9ec2f4`::moving from state preparing -> state finished >Thread-70666::DEBUG::2014-09-25 17:35:24,490::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70666::DEBUG::2014-09-25 17:35:24,490::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70666::DEBUG::2014-09-25 17:35:24,491::task::993::Storage.TaskManager.Task::(_decref) Task=`aa25b968-9153-4060-ab43-12c24f9ec2f4`::ref 0 aborting False >Thread-70666::DEBUG::2014-09-25 17:35:24,491::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70666::DEBUG::2014-09-25 17:35:24,491::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:24,508::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:24,509::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70667::DEBUG::2014-09-25 17:35:24,509::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70667::DEBUG::2014-09-25 17:35:24,511::task::595::Storage.TaskManager.Task::(_updateState) Task=`b7085321-8737-4ccc-91ef-9d0c278023b0`::moving from state init -> state preparing >Thread-70667::INFO::2014-09-25 17:35:24,511::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70667::DEBUG::2014-09-25 17:35:24,512::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5ca178a0-e1f9-42ee-8b9c-94c2a313426b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70667::DEBUG::2014-09-25 17:35:24,512::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70667::DEBUG::2014-09-25 17:35:24,512::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70667::DEBUG::2014-09-25 17:35:24,513::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5ca178a0-e1f9-42ee-8b9c-94c2a313426b`::Granted request >Thread-70667::DEBUG::2014-09-25 17:35:24,514::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b7085321-8737-4ccc-91ef-9d0c278023b0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70667::DEBUG::2014-09-25 17:35:24,514::task::993::Storage.TaskManager.Task::(_decref) Task=`b7085321-8737-4ccc-91ef-9d0c278023b0`::ref 1 aborting False >Thread-70667::INFO::2014-09-25 17:35:24,519::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70667::DEBUG::2014-09-25 17:35:24,519::task::1191::Storage.TaskManager.Task::(prepare) Task=`b7085321-8737-4ccc-91ef-9d0c278023b0`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70667::DEBUG::2014-09-25 17:35:24,519::task::595::Storage.TaskManager.Task::(_updateState) Task=`b7085321-8737-4ccc-91ef-9d0c278023b0`::moving from state preparing -> state finished >Thread-70667::DEBUG::2014-09-25 17:35:24,519::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70667::DEBUG::2014-09-25 17:35:24,520::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70667::DEBUG::2014-09-25 17:35:24,520::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70667::DEBUG::2014-09-25 17:35:24,520::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70667::DEBUG::2014-09-25 17:35:24,520::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70667::DEBUG::2014-09-25 17:35:24,521::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70667::DEBUG::2014-09-25 17:35:24,521::task::993::Storage.TaskManager.Task::(_decref) Task=`b7085321-8737-4ccc-91ef-9d0c278023b0`::ref 0 aborting False >Thread-70667::DEBUG::2014-09-25 17:35:24,521::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70667::DEBUG::2014-09-25 17:35:24,522::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:35:24,931::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:24,931::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:24,932::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:24,933::__init__::232::IOProcess::(_processLogs) (108716) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:35:24,933::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:35:24,934::__init__::232::IOProcess::(_processLogs) (108716) Queuing response >Thread-80::DEBUG::2014-09-25 17:35:24,943::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000548923 s, 689 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:24,945::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:24,945::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:24,945::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:24,945::__init__::232::IOProcess::(_processLogs) (108717) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:24,946::__init__::232::IOProcess::(_processLogs) (108717) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:25,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:25,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020149 s, 50.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:25,812::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:25,813::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70668::DEBUG::2014-09-25 17:35:25,816::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:35:26,901::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:26,901::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:26,901::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:26,902::__init__::232::IOProcess::(_processLogs) (108718) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:26,906::__init__::232::IOProcess::(_processLogs) (108718) Queuing response >Thread-79::DEBUG::2014-09-25 17:35:26,908::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:35:26,923::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0039664 s, 142 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:26,924::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:26,924::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:26,924::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:26,925::__init__::232::IOProcess::(_processLogs) (108719) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:26,926::__init__::232::IOProcess::(_processLogs) (108719) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:26,928::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:26,928::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:26,928::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:26,929::__init__::232::IOProcess::(_processLogs) (108720) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:26,929::__init__::232::IOProcess::(_processLogs) (108720) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:26,930::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:26,930::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:26,931::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:26,931::__init__::232::IOProcess::(_processLogs) (108721) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:26,931::__init__::232::IOProcess::(_processLogs) (108721) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:26,932::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:26,932::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:26,933::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:26,934::__init__::232::IOProcess::(_processLogs) (108722) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:26,934::__init__::232::IOProcess::(_processLogs) (108722) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:27,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:27,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164767 s, 62.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:28,834::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:28,836::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70669::DEBUG::2014-09-25 17:35:28,838::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:29,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:29,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142644 s, 71.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:35:31,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:31,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150878 s, 67.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:31,584::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:31,585::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:31,585::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:31,585::__init__::232::IOProcess::(_processLogs) (108723) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:31,590::__init__::232::IOProcess::(_processLogs) (108723) Queuing response >Thread-81::DEBUG::2014-09-25 17:35:31,592::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:35:31,608::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0061679 s, 60.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:31,610::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:31,610::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:31,610::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:31,610::__init__::232::IOProcess::(_processLogs) (108724) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:31,613::__init__::232::IOProcess::(_processLogs) (108724) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:31,859::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:31,861::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70670::DEBUG::2014-09-25 17:35:31,863::task::595::Storage.TaskManager.Task::(_updateState) Task=`f56dd737-e770-4851-aeca-3ee6b2cfa8cd`::moving from state init -> state preparing >Thread-70670::INFO::2014-09-25 17:35:31,864::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70670::INFO::2014-09-25 17:35:31,864::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0039664', 'lastCheck': '4.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000548923', 'lastCheck': '6.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0061679', 'lastCheck': '0.3', 'valid': True}} >Thread-70670::DEBUG::2014-09-25 17:35:31,864::task::1191::Storage.TaskManager.Task::(prepare) Task=`f56dd737-e770-4851-aeca-3ee6b2cfa8cd`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0039664', 'lastCheck': '4.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000548923', 'lastCheck': '6.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0061679', 'lastCheck': '0.3', 'valid': True}} >Thread-70670::DEBUG::2014-09-25 17:35:31,865::task::595::Storage.TaskManager.Task::(_updateState) Task=`f56dd737-e770-4851-aeca-3ee6b2cfa8cd`::moving from state preparing -> state finished >Thread-70670::DEBUG::2014-09-25 17:35:31,865::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70670::DEBUG::2014-09-25 17:35:31,865::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70670::DEBUG::2014-09-25 17:35:31,865::task::993::Storage.TaskManager.Task::(_decref) Task=`f56dd737-e770-4851-aeca-3ee6b2cfa8cd`::ref 0 aborting False >Thread-70670::DEBUG::2014-09-25 17:35:31,868::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:31,886::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:31,887::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70671::DEBUG::2014-09-25 17:35:31,889::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:33,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:33,395::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0176448 s, 58.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:34,551::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:34,553::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70672::DEBUG::2014-09-25 17:35:34,553::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70672::DEBUG::2014-09-25 17:35:34,555::task::595::Storage.TaskManager.Task::(_updateState) Task=`3f7e5a71-804f-4258-a507-53f714ebfe04`::moving from state init -> state preparing >Thread-70672::INFO::2014-09-25 17:35:34,555::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70672::INFO::2014-09-25 17:35:34,556::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70672::DEBUG::2014-09-25 17:35:34,556::task::1191::Storage.TaskManager.Task::(prepare) Task=`3f7e5a71-804f-4258-a507-53f714ebfe04`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70672::DEBUG::2014-09-25 17:35:34,556::task::595::Storage.TaskManager.Task::(_updateState) Task=`3f7e5a71-804f-4258-a507-53f714ebfe04`::moving from state preparing -> state finished >Thread-70672::DEBUG::2014-09-25 17:35:34,556::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70672::DEBUG::2014-09-25 17:35:34,556::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70672::DEBUG::2014-09-25 17:35:34,557::task::993::Storage.TaskManager.Task::(_decref) Task=`3f7e5a71-804f-4258-a507-53f714ebfe04`::ref 0 aborting False >Thread-70672::DEBUG::2014-09-25 17:35:34,557::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70672::DEBUG::2014-09-25 17:35:34,557::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:34,571::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70673::DEBUG::2014-09-25 17:35:34,572::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:35:34,572::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70673::DEBUG::2014-09-25 17:35:34,574::task::595::Storage.TaskManager.Task::(_updateState) Task=`71cc5a1d-c034-4fda-9bbf-1e04f9a29f6a`::moving from state init -> state preparing >Thread-70673::INFO::2014-09-25 17:35:34,574::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70673::DEBUG::2014-09-25 17:35:34,575::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`53681860-eb07-4737-b3eb-bdd278491e2c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70673::DEBUG::2014-09-25 17:35:34,575::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70673::DEBUG::2014-09-25 17:35:34,576::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70673::DEBUG::2014-09-25 17:35:34,576::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`53681860-eb07-4737-b3eb-bdd278491e2c`::Granted request >Thread-70673::DEBUG::2014-09-25 17:35:34,576::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`71cc5a1d-c034-4fda-9bbf-1e04f9a29f6a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70673::DEBUG::2014-09-25 17:35:34,577::task::993::Storage.TaskManager.Task::(_decref) Task=`71cc5a1d-c034-4fda-9bbf-1e04f9a29f6a`::ref 1 aborting False >Thread-70673::INFO::2014-09-25 17:35:34,581::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70673::DEBUG::2014-09-25 17:35:34,581::task::1191::Storage.TaskManager.Task::(prepare) Task=`71cc5a1d-c034-4fda-9bbf-1e04f9a29f6a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70673::DEBUG::2014-09-25 17:35:34,582::task::595::Storage.TaskManager.Task::(_updateState) Task=`71cc5a1d-c034-4fda-9bbf-1e04f9a29f6a`::moving from state preparing -> state finished >Thread-70673::DEBUG::2014-09-25 17:35:34,582::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70673::DEBUG::2014-09-25 17:35:34,582::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70673::DEBUG::2014-09-25 17:35:34,582::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70673::DEBUG::2014-09-25 17:35:34,583::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70673::DEBUG::2014-09-25 17:35:34,583::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70673::DEBUG::2014-09-25 17:35:34,583::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70673::DEBUG::2014-09-25 17:35:34,584::task::993::Storage.TaskManager.Task::(_decref) Task=`71cc5a1d-c034-4fda-9bbf-1e04f9a29f6a`::ref 0 aborting False >Thread-70673::DEBUG::2014-09-25 17:35:34,584::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70673::DEBUG::2014-09-25 17:35:34,584::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:35:34,948::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:34,949::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:34,949::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:34,949::__init__::232::IOProcess::(_processLogs) (108725) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:34,950::__init__::232::IOProcess::(_processLogs) (108725) Queuing response >Thread-80::DEBUG::2014-09-25 17:35:34,950::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:34,959::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:34,962::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-80::DEBUG::2014-09-25 17:35:34,963::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000692592 s, 546 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:34,966::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:34,969::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-70674::DEBUG::2014-09-25 17:35:34,969::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:35:34,971::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:34,973::__init__::232::IOProcess::(_processLogs) (108726) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:34,974::__init__::232::IOProcess::(_processLogs) (108726) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:35,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:35,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134171 s, 76.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:36,935::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:36,935::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:36,935::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:36,936::__init__::232::IOProcess::(_processLogs) (108727) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:36,937::__init__::232::IOProcess::(_processLogs) (108727) Queuing response >Thread-79::DEBUG::2014-09-25 17:35:36,938::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:35:36,955::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00364453 s, 154 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:36,956::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:36,956::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:36,956::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:36,957::__init__::232::IOProcess::(_processLogs) (108728) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:36,960::__init__::232::IOProcess::(_processLogs) (108728) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:36,962::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:36,962::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:36,963::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:36,963::__init__::232::IOProcess::(_processLogs) (108729) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:36,963::__init__::232::IOProcess::(_processLogs) (108729) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:36,964::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:36,964::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:36,965::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:36,965::__init__::232::IOProcess::(_processLogs) (108730) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:36,966::__init__::232::IOProcess::(_processLogs) (108730) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:36,966::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:36,967::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:36,968::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:36,968::__init__::232::IOProcess::(_processLogs) (108731) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:36,968::__init__::232::IOProcess::(_processLogs) (108731) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:37,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:37,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153596 s, 66.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:37,990::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:37,991::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70675::DEBUG::2014-09-25 17:35:37,993::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:39,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:39,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152359 s, 67.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:41,011::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:41,013::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70676::DEBUG::2014-09-25 17:35:41,014::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:41,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:41,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177439 s, 57.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:41,615::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:41,615::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:41,615::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:41,615::__init__::232::IOProcess::(_processLogs) (108732) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:41,621::__init__::232::IOProcess::(_processLogs) (108732) Queuing response >Thread-81::DEBUG::2014-09-25 17:35:41,622::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:35:41,641::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00428689 s, 86.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:41,642::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:41,642::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:41,643::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:41,643::__init__::232::IOProcess::(_processLogs) (108733) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:41,646::__init__::232::IOProcess::(_processLogs) (108733) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:43,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:43,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158216 s, 64.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:44,030::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:44,031::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70677::DEBUG::2014-09-25 17:35:44,033::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:44,629::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:44,630::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70678::DEBUG::2014-09-25 17:35:44,630::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70678::DEBUG::2014-09-25 17:35:44,632::task::595::Storage.TaskManager.Task::(_updateState) Task=`052e7c07-6843-43a1-8c24-3de02ad3386d`::moving from state init -> state preparing >Thread-70678::INFO::2014-09-25 17:35:44,633::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70678::INFO::2014-09-25 17:35:44,633::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70678::DEBUG::2014-09-25 17:35:44,634::task::1191::Storage.TaskManager.Task::(prepare) Task=`052e7c07-6843-43a1-8c24-3de02ad3386d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70678::DEBUG::2014-09-25 17:35:44,634::task::595::Storage.TaskManager.Task::(_updateState) Task=`052e7c07-6843-43a1-8c24-3de02ad3386d`::moving from state preparing -> state finished >Thread-70678::DEBUG::2014-09-25 17:35:44,634::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70678::DEBUG::2014-09-25 17:35:44,634::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70678::DEBUG::2014-09-25 17:35:44,634::task::993::Storage.TaskManager.Task::(_decref) Task=`052e7c07-6843-43a1-8c24-3de02ad3386d`::ref 0 aborting False >Thread-70678::DEBUG::2014-09-25 17:35:44,635::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70678::DEBUG::2014-09-25 17:35:44,635::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:44,649::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70679::DEBUG::2014-09-25 17:35:44,651::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:35:44,651::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70679::DEBUG::2014-09-25 17:35:44,653::task::595::Storage.TaskManager.Task::(_updateState) Task=`f8633d2c-2272-4b7d-b8d3-08d66f148fb3`::moving from state init -> state preparing >Thread-70679::INFO::2014-09-25 17:35:44,653::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70679::DEBUG::2014-09-25 17:35:44,654::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2b66a2fb-a43b-4c56-ab61-d8058e85c818`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70679::DEBUG::2014-09-25 17:35:44,654::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70679::DEBUG::2014-09-25 17:35:44,655::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70679::DEBUG::2014-09-25 17:35:44,655::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2b66a2fb-a43b-4c56-ab61-d8058e85c818`::Granted request >Thread-70679::DEBUG::2014-09-25 17:35:44,656::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f8633d2c-2272-4b7d-b8d3-08d66f148fb3`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70679::DEBUG::2014-09-25 17:35:44,656::task::993::Storage.TaskManager.Task::(_decref) Task=`f8633d2c-2272-4b7d-b8d3-08d66f148fb3`::ref 1 aborting False >Thread-70679::INFO::2014-09-25 17:35:44,660::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70679::DEBUG::2014-09-25 17:35:44,661::task::1191::Storage.TaskManager.Task::(prepare) Task=`f8633d2c-2272-4b7d-b8d3-08d66f148fb3`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70679::DEBUG::2014-09-25 17:35:44,661::task::595::Storage.TaskManager.Task::(_updateState) Task=`f8633d2c-2272-4b7d-b8d3-08d66f148fb3`::moving from state preparing -> state finished >Thread-70679::DEBUG::2014-09-25 17:35:44,661::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70679::DEBUG::2014-09-25 17:35:44,662::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70679::DEBUG::2014-09-25 17:35:44,662::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70679::DEBUG::2014-09-25 17:35:44,662::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70679::DEBUG::2014-09-25 17:35:44,662::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70679::DEBUG::2014-09-25 17:35:44,663::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70679::DEBUG::2014-09-25 17:35:44,663::task::993::Storage.TaskManager.Task::(_decref) Task=`f8633d2c-2272-4b7d-b8d3-08d66f148fb3`::ref 0 aborting False >Thread-70679::DEBUG::2014-09-25 17:35:44,664::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70679::DEBUG::2014-09-25 17:35:44,664::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:35:44,972::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:44,973::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:44,973::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:44,973::__init__::232::IOProcess::(_processLogs) (108734) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:44,974::__init__::232::IOProcess::(_processLogs) (108734) Queuing response >Thread-80::DEBUG::2014-09-25 17:35:44,974::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:35:44,984::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000488628 s, 774 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:44,985::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:44,985::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:44,986::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:44,988::__init__::232::IOProcess::(_processLogs) (108735) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:44,988::__init__::232::IOProcess::(_processLogs) (108735) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:45,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:45,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163439 s, 62.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:46,969::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:46,970::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:46,971::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:46,971::__init__::232::IOProcess::(_processLogs) (108736) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:46,973::__init__::232::IOProcess::(_processLogs) (108736) Queuing response >Thread-79::DEBUG::2014-09-25 17:35:46,974::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:35:46,987::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00353291 s, 159 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:46,990::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:46,990::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:46,991::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:46,993::__init__::232::IOProcess::(_processLogs) (108737) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:46,994::__init__::232::IOProcess::(_processLogs) (108737) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:46,995::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:46,996::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:46,996::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:46,997::__init__::232::IOProcess::(_processLogs) (108738) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:46,998::__init__::232::IOProcess::(_processLogs) (108738) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:47,000::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:47,001::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:47,001::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:47,002::__init__::232::IOProcess::(_processLogs) (108739) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:47,006::__init__::232::IOProcess::(_processLogs) (108739) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:47,007::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:47,008::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:47,008::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:47,009::__init__::232::IOProcess::(_processLogs) (108740) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:47,014::__init__::232::IOProcess::(_processLogs) (108740) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:47,047::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:47,048::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70680::DEBUG::2014-09-25 17:35:47,050::task::595::Storage.TaskManager.Task::(_updateState) Task=`b8048c73-b68a-4b9b-bf0b-1844e2efa536`::moving from state init -> state preparing >Thread-70680::INFO::2014-09-25 17:35:47,050::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70680::INFO::2014-09-25 17:35:47,051::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00353291', 'lastCheck': '0.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000488628', 'lastCheck': '2.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00428689', 'lastCheck': '5.4', 'valid': True}} >Thread-70680::DEBUG::2014-09-25 17:35:47,051::task::1191::Storage.TaskManager.Task::(prepare) Task=`b8048c73-b68a-4b9b-bf0b-1844e2efa536`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00353291', 'lastCheck': '0.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000488628', 'lastCheck': '2.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00428689', 'lastCheck': '5.4', 'valid': True}} >Thread-70680::DEBUG::2014-09-25 17:35:47,051::task::595::Storage.TaskManager.Task::(_updateState) Task=`b8048c73-b68a-4b9b-bf0b-1844e2efa536`::moving from state preparing -> state finished >Thread-70680::DEBUG::2014-09-25 17:35:47,051::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70680::DEBUG::2014-09-25 17:35:47,051::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70680::DEBUG::2014-09-25 17:35:47,051::task::993::Storage.TaskManager.Task::(_decref) Task=`b8048c73-b68a-4b9b-bf0b-1844e2efa536`::ref 0 aborting False >Thread-70680::DEBUG::2014-09-25 17:35:47,053::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:47,069::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:47,069::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70681::DEBUG::2014-09-25 17:35:47,071::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:47,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:47,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186198 s, 55.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:35:49,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:49,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0294081 s, 34.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:50,118::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:50,119::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70682::DEBUG::2014-09-25 17:35:50,121::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:35:51,648::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:51,649::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:51,649::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:51,649::__init__::232::IOProcess::(_processLogs) (108741) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:51,649::__init__::232::IOProcess::(_processLogs) (108741) Queuing response >Thread-81::DEBUG::2014-09-25 17:35:51,651::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:35:51,661::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00111602 s, 332 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:51,663::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:51,663::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:51,663::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:51,664::__init__::232::IOProcess::(_processLogs) (108742) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:51,664::__init__::232::IOProcess::(_processLogs) (108742) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:51,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:51,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155927 s, 65.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:53,139::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:53,141::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70683::DEBUG::2014-09-25 17:35:53,143::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:35:53,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:53,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191914 s, 53.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:54,692::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70684::DEBUG::2014-09-25 17:35:54,693::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:35:54,693::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70684::DEBUG::2014-09-25 17:35:54,696::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d93bba4-d2a1-4070-bbe0-11e654ddd4eb`::moving from state init -> state preparing >Thread-70684::INFO::2014-09-25 17:35:54,696::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70684::INFO::2014-09-25 17:35:54,697::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70684::DEBUG::2014-09-25 17:35:54,697::task::1191::Storage.TaskManager.Task::(prepare) Task=`7d93bba4-d2a1-4070-bbe0-11e654ddd4eb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70684::DEBUG::2014-09-25 17:35:54,697::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d93bba4-d2a1-4070-bbe0-11e654ddd4eb`::moving from state preparing -> state finished >Thread-70684::DEBUG::2014-09-25 17:35:54,697::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70684::DEBUG::2014-09-25 17:35:54,698::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70684::DEBUG::2014-09-25 17:35:54,698::task::993::Storage.TaskManager.Task::(_decref) Task=`7d93bba4-d2a1-4070-bbe0-11e654ddd4eb`::ref 0 aborting False >Thread-70684::DEBUG::2014-09-25 17:35:54,698::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70684::DEBUG::2014-09-25 17:35:54,699::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:54,713::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:54,714::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70685::DEBUG::2014-09-25 17:35:54,715::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70685::DEBUG::2014-09-25 17:35:54,716::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3e13bfb-d070-4610-bd22-cc8d086c86d7`::moving from state init -> state preparing >Thread-70685::INFO::2014-09-25 17:35:54,716::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70685::DEBUG::2014-09-25 17:35:54,717::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`80a865c9-966e-4c05-b13c-4a2e00e1991c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70685::DEBUG::2014-09-25 17:35:54,717::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70685::DEBUG::2014-09-25 17:35:54,718::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70685::DEBUG::2014-09-25 17:35:54,718::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`80a865c9-966e-4c05-b13c-4a2e00e1991c`::Granted request >Thread-70685::DEBUG::2014-09-25 17:35:54,719::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e3e13bfb-d070-4610-bd22-cc8d086c86d7`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70685::DEBUG::2014-09-25 17:35:54,719::task::993::Storage.TaskManager.Task::(_decref) Task=`e3e13bfb-d070-4610-bd22-cc8d086c86d7`::ref 1 aborting False >Thread-70685::INFO::2014-09-25 17:35:54,723::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70685::DEBUG::2014-09-25 17:35:54,724::task::1191::Storage.TaskManager.Task::(prepare) Task=`e3e13bfb-d070-4610-bd22-cc8d086c86d7`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70685::DEBUG::2014-09-25 17:35:54,724::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3e13bfb-d070-4610-bd22-cc8d086c86d7`::moving from state preparing -> state finished >Thread-70685::DEBUG::2014-09-25 17:35:54,724::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70685::DEBUG::2014-09-25 17:35:54,724::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70685::DEBUG::2014-09-25 17:35:54,725::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70685::DEBUG::2014-09-25 17:35:54,725::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70685::DEBUG::2014-09-25 17:35:54,725::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70685::DEBUG::2014-09-25 17:35:54,725::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70685::DEBUG::2014-09-25 17:35:54,726::task::993::Storage.TaskManager.Task::(_decref) Task=`e3e13bfb-d070-4610-bd22-cc8d086c86d7`::ref 0 aborting False >Thread-70685::DEBUG::2014-09-25 17:35:54,726::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70685::DEBUG::2014-09-25 17:35:54,727::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:35:54,988::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:54,989::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:54,989::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:54,989::__init__::232::IOProcess::(_processLogs) (108743) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:54,990::__init__::232::IOProcess::(_processLogs) (108743) Queuing response >Thread-80::DEBUG::2014-09-25 17:35:54,991::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:35:55,000::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000559078 s, 676 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:55,001::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:55,001::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:55,002::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:55,003::__init__::232::IOProcess::(_processLogs) (108744) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:55,004::__init__::232::IOProcess::(_processLogs) (108744) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:55,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:55,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148827 s, 68.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:56,160::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:56,162::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70686::DEBUG::2014-09-25 17:35:56,164::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:35:57,015::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:57,016::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:57,016::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:57,016::__init__::232::IOProcess::(_processLogs) (108745) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:57,024::__init__::232::IOProcess::(_processLogs) (108745) Queuing response >Thread-79::DEBUG::2014-09-25 17:35:57,025::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:35:57,044::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00306273 s, 183 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:35:57,045::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:57,045::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:57,046::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:57,046::__init__::232::IOProcess::(_processLogs) (108746) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:35:57,053::__init__::232::IOProcess::(_processLogs) (108746) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:57,054::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:57,055::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:57,055::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:57,055::__init__::232::IOProcess::(_processLogs) (108747) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:57,056::__init__::232::IOProcess::(_processLogs) (108747) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:57,057::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:57,057::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:57,057::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:57,058::__init__::232::IOProcess::(_processLogs) (108748) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:57,058::__init__::232::IOProcess::(_processLogs) (108748) Queuing response >Thread-29::DEBUG::2014-09-25 17:35:57,059::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:35:57,059::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:35:57,059::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:35:57,061::__init__::232::IOProcess::(_processLogs) (108749) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:35:57,061::__init__::232::IOProcess::(_processLogs) (108749) Queuing response >Dummy-148::DEBUG::2014-09-25 17:35:57,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:35:58,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193148 s, 53.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:35:59,179::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:35:59,181::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70687::DEBUG::2014-09-25 17:35:59,183::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:00,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:00,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135247 s, 75.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:01,667::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:01,667::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:01,668::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:01,668::__init__::232::IOProcess::(_processLogs) (108750) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:01,668::__init__::232::IOProcess::(_processLogs) (108750) Queuing response >Thread-81::DEBUG::2014-09-25 17:36:01,669::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:36:01,679::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000455948 s, 814 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:01,680::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:01,680::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:01,681::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:01,682::__init__::232::IOProcess::(_processLogs) (108751) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:01,682::__init__::232::IOProcess::(_processLogs) (108751) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:02,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:02,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199558 s, 51.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:02,199::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:02,200::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70688::DEBUG::2014-09-25 17:36:02,203::task::595::Storage.TaskManager.Task::(_updateState) Task=`54f6319e-a341-4244-834d-7b9a1b4b3c18`::moving from state init -> state preparing >Thread-70688::INFO::2014-09-25 17:36:02,203::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70688::INFO::2014-09-25 17:36:02,204::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00306273', 'lastCheck': '5.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000559078', 'lastCheck': '7.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000455948', 'lastCheck': '0.5', 'valid': True}} >Thread-70688::DEBUG::2014-09-25 17:36:02,204::task::1191::Storage.TaskManager.Task::(prepare) Task=`54f6319e-a341-4244-834d-7b9a1b4b3c18`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00306273', 'lastCheck': '5.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000559078', 'lastCheck': '7.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000455948', 'lastCheck': '0.5', 'valid': True}} >Thread-70688::DEBUG::2014-09-25 17:36:02,204::task::595::Storage.TaskManager.Task::(_updateState) Task=`54f6319e-a341-4244-834d-7b9a1b4b3c18`::moving from state preparing -> state finished >Thread-70688::DEBUG::2014-09-25 17:36:02,205::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70688::DEBUG::2014-09-25 17:36:02,205::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70688::DEBUG::2014-09-25 17:36:02,205::task::993::Storage.TaskManager.Task::(_decref) Task=`54f6319e-a341-4244-834d-7b9a1b4b3c18`::ref 0 aborting False >Thread-70688::DEBUG::2014-09-25 17:36:02,207::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:02,229::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:02,230::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70689::DEBUG::2014-09-25 17:36:02,232::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:04,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:04,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213069 s, 48.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:04,768::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70690::DEBUG::2014-09-25 17:36:04,769::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:36:04,769::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70690::DEBUG::2014-09-25 17:36:04,771::task::595::Storage.TaskManager.Task::(_updateState) Task=`22790a28-963f-4c1f-a734-46a149c6f0dd`::moving from state init -> state preparing >Thread-70690::INFO::2014-09-25 17:36:04,772::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70690::INFO::2014-09-25 17:36:04,772::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70690::DEBUG::2014-09-25 17:36:04,773::task::1191::Storage.TaskManager.Task::(prepare) Task=`22790a28-963f-4c1f-a734-46a149c6f0dd`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70690::DEBUG::2014-09-25 17:36:04,773::task::595::Storage.TaskManager.Task::(_updateState) Task=`22790a28-963f-4c1f-a734-46a149c6f0dd`::moving from state preparing -> state finished >Thread-70690::DEBUG::2014-09-25 17:36:04,773::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70690::DEBUG::2014-09-25 17:36:04,773::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70690::DEBUG::2014-09-25 17:36:04,773::task::993::Storage.TaskManager.Task::(_decref) Task=`22790a28-963f-4c1f-a734-46a149c6f0dd`::ref 0 aborting False >Thread-70690::DEBUG::2014-09-25 17:36:04,774::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70690::DEBUG::2014-09-25 17:36:04,774::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:04,800::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70691::DEBUG::2014-09-25 17:36:04,801::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:36:04,801::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70691::DEBUG::2014-09-25 17:36:04,803::task::595::Storage.TaskManager.Task::(_updateState) Task=`2970dd78-4f2d-4296-9d42-8cfe44857521`::moving from state init -> state preparing >Thread-70691::INFO::2014-09-25 17:36:04,803::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70691::DEBUG::2014-09-25 17:36:04,804::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`213ebead-5640-465e-95fd-3033157048f2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70691::DEBUG::2014-09-25 17:36:04,804::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70691::DEBUG::2014-09-25 17:36:04,805::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70691::DEBUG::2014-09-25 17:36:04,805::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`213ebead-5640-465e-95fd-3033157048f2`::Granted request >Thread-70691::DEBUG::2014-09-25 17:36:04,805::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2970dd78-4f2d-4296-9d42-8cfe44857521`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70691::DEBUG::2014-09-25 17:36:04,806::task::993::Storage.TaskManager.Task::(_decref) Task=`2970dd78-4f2d-4296-9d42-8cfe44857521`::ref 1 aborting False >Thread-70691::INFO::2014-09-25 17:36:04,810::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70691::DEBUG::2014-09-25 17:36:04,810::task::1191::Storage.TaskManager.Task::(prepare) Task=`2970dd78-4f2d-4296-9d42-8cfe44857521`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70691::DEBUG::2014-09-25 17:36:04,811::task::595::Storage.TaskManager.Task::(_updateState) Task=`2970dd78-4f2d-4296-9d42-8cfe44857521`::moving from state preparing -> state finished >Thread-70691::DEBUG::2014-09-25 17:36:04,811::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70691::DEBUG::2014-09-25 17:36:04,811::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70691::DEBUG::2014-09-25 17:36:04,811::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70691::DEBUG::2014-09-25 17:36:04,812::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70691::DEBUG::2014-09-25 17:36:04,812::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70691::DEBUG::2014-09-25 17:36:04,812::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70691::DEBUG::2014-09-25 17:36:04,813::task::993::Storage.TaskManager.Task::(_decref) Task=`2970dd78-4f2d-4296-9d42-8cfe44857521`::ref 0 aborting False >Thread-70691::DEBUG::2014-09-25 17:36:04,813::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70691::DEBUG::2014-09-25 17:36:04,814::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:36:05,004::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:05,004::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:05,005::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:05,005::__init__::232::IOProcess::(_processLogs) (108752) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:05,005::__init__::232::IOProcess::(_processLogs) (108752) Queuing response >Thread-80::DEBUG::2014-09-25 17:36:05,007::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:36:05,016::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000471028 s, 803 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:05,017::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:05,017::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:05,017::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:05,017::__init__::232::IOProcess::(_processLogs) (108753) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:05,018::__init__::232::IOProcess::(_processLogs) (108753) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:05,291::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:05,292::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70692::DEBUG::2014-09-25 17:36:05,294::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:06,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:06,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131304 s, 78.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:07,062::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:07,062::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:07,062::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:07,062::__init__::232::IOProcess::(_processLogs) (108754) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:07,066::__init__::232::IOProcess::(_processLogs) (108754) Queuing response >Thread-79::DEBUG::2014-09-25 17:36:07,067::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:36:07,086::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00367868 s, 153 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:07,087::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:07,088::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:07,088::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:07,088::__init__::232::IOProcess::(_processLogs) (108755) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:07,097::__init__::232::IOProcess::(_processLogs) (108755) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:07,098::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:07,098::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:07,099::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:07,099::__init__::232::IOProcess::(_processLogs) (108756) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:07,099::__init__::232::IOProcess::(_processLogs) (108756) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:07,100::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:07,101::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:07,101::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:07,101::__init__::232::IOProcess::(_processLogs) (108757) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:07,102::__init__::232::IOProcess::(_processLogs) (108757) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:07,102::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:07,103::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:07,104::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:07,104::__init__::232::IOProcess::(_processLogs) (108758) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:07,104::__init__::232::IOProcess::(_processLogs) (108758) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:08,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:08,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131482 s, 77.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:08,314::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:08,315::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70693::DEBUG::2014-09-25 17:36:08,317::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:10,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:10,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.217199 s, 4.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:11,333::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:11,334::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70694::DEBUG::2014-09-25 17:36:11,336::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:36:11,684::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:11,684::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:11,685::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:11,686::__init__::232::IOProcess::(_processLogs) (108759) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:11,689::__init__::232::IOProcess::(_processLogs) (108759) Queuing response >Thread-81::DEBUG::2014-09-25 17:36:11,691::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:36:11,706::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00339793 s, 109 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:11,707::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:11,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:11,708::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:11,708::__init__::232::IOProcess::(_processLogs) (108760) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:11,711::__init__::232::IOProcess::(_processLogs) (108760) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:12,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:12,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181687 s, 56.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:14,351::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:14,352::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70695::DEBUG::2014-09-25 17:36:14,354::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:14,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:14,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150111 s, 68.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:14,857::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:14,858::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70696::DEBUG::2014-09-25 17:36:14,858::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70696::DEBUG::2014-09-25 17:36:14,860::task::595::Storage.TaskManager.Task::(_updateState) Task=`be40c55f-c762-4b55-beac-d5beccd9f05a`::moving from state init -> state preparing >Thread-70696::INFO::2014-09-25 17:36:14,860::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70696::INFO::2014-09-25 17:36:14,861::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70696::DEBUG::2014-09-25 17:36:14,861::task::1191::Storage.TaskManager.Task::(prepare) Task=`be40c55f-c762-4b55-beac-d5beccd9f05a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70696::DEBUG::2014-09-25 17:36:14,861::task::595::Storage.TaskManager.Task::(_updateState) Task=`be40c55f-c762-4b55-beac-d5beccd9f05a`::moving from state preparing -> state finished >Thread-70696::DEBUG::2014-09-25 17:36:14,862::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70696::DEBUG::2014-09-25 17:36:14,862::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70696::DEBUG::2014-09-25 17:36:14,862::task::993::Storage.TaskManager.Task::(_decref) Task=`be40c55f-c762-4b55-beac-d5beccd9f05a`::ref 0 aborting False >Thread-70696::DEBUG::2014-09-25 17:36:14,863::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70696::DEBUG::2014-09-25 17:36:14,863::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:14,873::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70697::DEBUG::2014-09-25 17:36:14,875::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:36:14,875::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70697::DEBUG::2014-09-25 17:36:14,877::task::595::Storage.TaskManager.Task::(_updateState) Task=`9650d003-ca1a-4789-99d0-5221947da71f`::moving from state init -> state preparing >Thread-70697::INFO::2014-09-25 17:36:14,877::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70697::DEBUG::2014-09-25 17:36:14,878::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`937856bb-9afa-4d37-a323-405eab456419`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70697::DEBUG::2014-09-25 17:36:14,878::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70697::DEBUG::2014-09-25 17:36:14,879::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70697::DEBUG::2014-09-25 17:36:14,879::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`937856bb-9afa-4d37-a323-405eab456419`::Granted request >Thread-70697::DEBUG::2014-09-25 17:36:14,880::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9650d003-ca1a-4789-99d0-5221947da71f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70697::DEBUG::2014-09-25 17:36:14,880::task::993::Storage.TaskManager.Task::(_decref) Task=`9650d003-ca1a-4789-99d0-5221947da71f`::ref 1 aborting False >Thread-70697::INFO::2014-09-25 17:36:14,884::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70697::DEBUG::2014-09-25 17:36:14,885::task::1191::Storage.TaskManager.Task::(prepare) Task=`9650d003-ca1a-4789-99d0-5221947da71f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70697::DEBUG::2014-09-25 17:36:14,885::task::595::Storage.TaskManager.Task::(_updateState) Task=`9650d003-ca1a-4789-99d0-5221947da71f`::moving from state preparing -> state finished >Thread-70697::DEBUG::2014-09-25 17:36:14,885::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70697::DEBUG::2014-09-25 17:36:14,885::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70697::DEBUG::2014-09-25 17:36:14,886::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70697::DEBUG::2014-09-25 17:36:14,886::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70697::DEBUG::2014-09-25 17:36:14,886::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70697::DEBUG::2014-09-25 17:36:14,886::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70697::DEBUG::2014-09-25 17:36:14,887::task::993::Storage.TaskManager.Task::(_decref) Task=`9650d003-ca1a-4789-99d0-5221947da71f`::ref 0 aborting False >Thread-70697::DEBUG::2014-09-25 17:36:14,887::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70697::DEBUG::2014-09-25 17:36:14,888::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:36:15,020::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:15,020::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:15,021::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:15,021::__init__::232::IOProcess::(_processLogs) (108761) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:15,021::__init__::232::IOProcess::(_processLogs) (108761) Queuing response >Thread-80::DEBUG::2014-09-25 17:36:15,023::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:36:15,031::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000384705 s, 983 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:15,032::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:15,033::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:15,033::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:15,033::__init__::232::IOProcess::(_processLogs) (108762) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:15,034::__init__::232::IOProcess::(_processLogs) (108762) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:16,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:16,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0314687 s, 32.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:17,104::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:17,105::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:17,105::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:17,105::__init__::232::IOProcess::(_processLogs) (108763) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:17,106::__init__::232::IOProcess::(_processLogs) (108763) Queuing response >Thread-79::DEBUG::2014-09-25 17:36:17,107::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:36:17,115::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000361986 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:17,116::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:17,116::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:17,117::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:17,117::__init__::232::IOProcess::(_processLogs) (108764) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:17,117::__init__::232::IOProcess::(_processLogs) (108764) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:17,119::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:17,119::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:17,119::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:17,120::__init__::232::IOProcess::(_processLogs) (108765) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:17,120::__init__::232::IOProcess::(_processLogs) (108765) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:17,121::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:17,121::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:17,121::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:17,122::__init__::232::IOProcess::(_processLogs) (108766) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:17,122::__init__::232::IOProcess::(_processLogs) (108766) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:17,123::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:17,123::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:17,123::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:17,124::__init__::232::IOProcess::(_processLogs) (108767) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:17,124::__init__::232::IOProcess::(_processLogs) (108767) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:17,367::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:17,369::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70698::DEBUG::2014-09-25 17:36:17,371::task::595::Storage.TaskManager.Task::(_updateState) Task=`72d5dbf0-f0d5-4133-a9ae-88fdcb469f5e`::moving from state init -> state preparing >Thread-70698::INFO::2014-09-25 17:36:17,371::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70698::INFO::2014-09-25 17:36:17,372::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000361986', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000384705', 'lastCheck': '2.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00339793', 'lastCheck': '5.7', 'valid': True}} >Thread-70698::DEBUG::2014-09-25 17:36:17,372::task::1191::Storage.TaskManager.Task::(prepare) Task=`72d5dbf0-f0d5-4133-a9ae-88fdcb469f5e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000361986', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000384705', 'lastCheck': '2.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00339793', 'lastCheck': '5.7', 'valid': True}} >Thread-70698::DEBUG::2014-09-25 17:36:17,372::task::595::Storage.TaskManager.Task::(_updateState) Task=`72d5dbf0-f0d5-4133-a9ae-88fdcb469f5e`::moving from state preparing -> state finished >Thread-70698::DEBUG::2014-09-25 17:36:17,372::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70698::DEBUG::2014-09-25 17:36:17,373::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70698::DEBUG::2014-09-25 17:36:17,373::task::993::Storage.TaskManager.Task::(_decref) Task=`72d5dbf0-f0d5-4133-a9ae-88fdcb469f5e`::ref 0 aborting False >Thread-70698::DEBUG::2014-09-25 17:36:17,375::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:17,390::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:17,391::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70699::DEBUG::2014-09-25 17:36:17,392::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:18,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:18,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141019 s, 72.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:20,438::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:20,440::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70700::DEBUG::2014-09-25 17:36:20,441::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:20,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:20,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0195663 s, 52.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:21,714::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:21,714::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:21,715::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:36:21,716::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:36:21,716::__init__::232::IOProcess::(_processLogs) (108768) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:21,726::__init__::232::IOProcess::(_processLogs) (108768) Queuing response >Thread-81::DEBUG::2014-09-25 17:36:21,728::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000428058 s, 867 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:21,731::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:21,733::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:21,734::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:21,735::__init__::232::IOProcess::(_processLogs) (108769) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:21,737::__init__::232::IOProcess::(_processLogs) (108769) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:22,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:22,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179298 s, 57.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:23,455::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:23,457::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70701::DEBUG::2014-09-25 17:36:23,459::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:24,911::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70702::DEBUG::2014-09-25 17:36:24,912::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:36:24,912::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70702::DEBUG::2014-09-25 17:36:24,914::task::595::Storage.TaskManager.Task::(_updateState) Task=`bbee0e19-62a9-48a0-9b18-0b963042b4d2`::moving from state init -> state preparing >Thread-70702::INFO::2014-09-25 17:36:24,914::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70702::INFO::2014-09-25 17:36:24,915::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70702::DEBUG::2014-09-25 17:36:24,915::task::1191::Storage.TaskManager.Task::(prepare) Task=`bbee0e19-62a9-48a0-9b18-0b963042b4d2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70702::DEBUG::2014-09-25 17:36:24,915::task::595::Storage.TaskManager.Task::(_updateState) Task=`bbee0e19-62a9-48a0-9b18-0b963042b4d2`::moving from state preparing -> state finished >Thread-70702::DEBUG::2014-09-25 17:36:24,916::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70702::DEBUG::2014-09-25 17:36:24,916::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70702::DEBUG::2014-09-25 17:36:24,916::task::993::Storage.TaskManager.Task::(_decref) Task=`bbee0e19-62a9-48a0-9b18-0b963042b4d2`::ref 0 aborting False >Thread-70702::DEBUG::2014-09-25 17:36:24,916::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70702::DEBUG::2014-09-25 17:36:24,917::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:24,929::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:24,931::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70703::DEBUG::2014-09-25 17:36:24,931::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70703::DEBUG::2014-09-25 17:36:24,932::task::595::Storage.TaskManager.Task::(_updateState) Task=`34a76915-d324-4d53-a896-305885839326`::moving from state init -> state preparing >Thread-70703::INFO::2014-09-25 17:36:24,933::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70703::DEBUG::2014-09-25 17:36:24,933::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ef1c2a9b-b142-4df0-b31c-2bff251931a1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70703::DEBUG::2014-09-25 17:36:24,934::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70703::DEBUG::2014-09-25 17:36:24,934::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70703::DEBUG::2014-09-25 17:36:24,934::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ef1c2a9b-b142-4df0-b31c-2bff251931a1`::Granted request >Thread-70703::DEBUG::2014-09-25 17:36:24,935::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`34a76915-d324-4d53-a896-305885839326`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70703::DEBUG::2014-09-25 17:36:24,935::task::993::Storage.TaskManager.Task::(_decref) Task=`34a76915-d324-4d53-a896-305885839326`::ref 1 aborting False >Thread-70703::INFO::2014-09-25 17:36:24,939::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70703::DEBUG::2014-09-25 17:36:24,940::task::1191::Storage.TaskManager.Task::(prepare) Task=`34a76915-d324-4d53-a896-305885839326`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70703::DEBUG::2014-09-25 17:36:24,940::task::595::Storage.TaskManager.Task::(_updateState) Task=`34a76915-d324-4d53-a896-305885839326`::moving from state preparing -> state finished >Thread-70703::DEBUG::2014-09-25 17:36:24,940::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70703::DEBUG::2014-09-25 17:36:24,940::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70703::DEBUG::2014-09-25 17:36:24,941::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70703::DEBUG::2014-09-25 17:36:24,941::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70703::DEBUG::2014-09-25 17:36:24,941::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70703::DEBUG::2014-09-25 17:36:24,941::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70703::DEBUG::2014-09-25 17:36:24,942::task::993::Storage.TaskManager.Task::(_decref) Task=`34a76915-d324-4d53-a896-305885839326`::ref 0 aborting False >Thread-70703::DEBUG::2014-09-25 17:36:24,942::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70703::DEBUG::2014-09-25 17:36:24,943::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:24,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:24,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0139597 s, 73.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:25,036::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:25,037::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:25,037::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:25,037::__init__::232::IOProcess::(_processLogs) (108770) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:25,038::__init__::232::IOProcess::(_processLogs) (108770) Queuing response >Thread-80::DEBUG::2014-09-25 17:36:25,039::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:36:25,047::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000439106 s, 861 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:25,048::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:25,049::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:25,049::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:25,049::__init__::232::IOProcess::(_processLogs) (108771) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:25,049::__init__::232::IOProcess::(_processLogs) (108771) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:26,475::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:26,476::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70704::DEBUG::2014-09-25 17:36:26,478::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:27,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:27,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0262897 s, 39.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:27,125::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:27,125::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:27,126::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:27,126::__init__::232::IOProcess::(_processLogs) (108772) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:27,129::__init__::232::IOProcess::(_processLogs) (108772) Queuing response >Thread-79::DEBUG::2014-09-25 17:36:27,130::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:36:27,151::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00410078 s, 137 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:27,152::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:27,153::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:27,153::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:27,153::__init__::232::IOProcess::(_processLogs) (108773) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:27,156::__init__::232::IOProcess::(_processLogs) (108773) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:27,157::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:27,157::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:27,158::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:27,158::__init__::232::IOProcess::(_processLogs) (108774) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:27,158::__init__::232::IOProcess::(_processLogs) (108774) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:27,159::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:27,160::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:27,160::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:27,160::__init__::232::IOProcess::(_processLogs) (108775) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:27,161::__init__::232::IOProcess::(_processLogs) (108775) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:27,161::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:27,161::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:27,162::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:27,162::__init__::232::IOProcess::(_processLogs) (108776) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:27,162::__init__::232::IOProcess::(_processLogs) (108776) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:29,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:29,105::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143981 s, 71.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:29,498::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:29,499::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70705::DEBUG::2014-09-25 17:36:29,501::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:31,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:31,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205957 s, 49.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:31,740::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:31,740::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:31,741::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:31,741::__init__::232::IOProcess::(_processLogs) (108777) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:31,792::__init__::232::IOProcess::(_processLogs) (108777) Queuing response >Thread-81::DEBUG::2014-09-25 17:36:31,793::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:36:31,816::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00439141 s, 84.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:31,817::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:31,817::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:31,817::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:31,818::__init__::232::IOProcess::(_processLogs) (108778) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:31,825::__init__::232::IOProcess::(_processLogs) (108778) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:32,515::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:32,516::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70706::DEBUG::2014-09-25 17:36:32,518::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5835c41-8001-4ebe-9080-2b8c1a6aea5b`::moving from state init -> state preparing >Thread-70706::INFO::2014-09-25 17:36:32,519::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70706::INFO::2014-09-25 17:36:32,519::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00410078', 'lastCheck': '5.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000439106', 'lastCheck': '7.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00439141', 'lastCheck': '0.7', 'valid': True}} >Thread-70706::DEBUG::2014-09-25 17:36:32,520::task::1191::Storage.TaskManager.Task::(prepare) Task=`f5835c41-8001-4ebe-9080-2b8c1a6aea5b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00410078', 'lastCheck': '5.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000439106', 'lastCheck': '7.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00439141', 'lastCheck': '0.7', 'valid': True}} >Thread-70706::DEBUG::2014-09-25 17:36:32,520::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5835c41-8001-4ebe-9080-2b8c1a6aea5b`::moving from state preparing -> state finished >Thread-70706::DEBUG::2014-09-25 17:36:32,520::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70706::DEBUG::2014-09-25 17:36:32,520::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70706::DEBUG::2014-09-25 17:36:32,520::task::993::Storage.TaskManager.Task::(_decref) Task=`f5835c41-8001-4ebe-9080-2b8c1a6aea5b`::ref 0 aborting False >Thread-70706::DEBUG::2014-09-25 17:36:32,523::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:32,539::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:32,540::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70707::DEBUG::2014-09-25 17:36:32,542::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:33,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:33,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175079 s, 58.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:34,969::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70708::DEBUG::2014-09-25 17:36:34,970::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:36:34,970::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70708::DEBUG::2014-09-25 17:36:34,973::task::595::Storage.TaskManager.Task::(_updateState) Task=`34c31c95-f079-4bc9-9d64-f6cd6f2d8518`::moving from state init -> state preparing >Thread-70708::INFO::2014-09-25 17:36:34,973::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70708::INFO::2014-09-25 17:36:34,974::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70708::DEBUG::2014-09-25 17:36:34,974::task::1191::Storage.TaskManager.Task::(prepare) Task=`34c31c95-f079-4bc9-9d64-f6cd6f2d8518`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70708::DEBUG::2014-09-25 17:36:34,974::task::595::Storage.TaskManager.Task::(_updateState) Task=`34c31c95-f079-4bc9-9d64-f6cd6f2d8518`::moving from state preparing -> state finished >Thread-70708::DEBUG::2014-09-25 17:36:34,974::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70708::DEBUG::2014-09-25 17:36:34,975::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70708::DEBUG::2014-09-25 17:36:34,975::task::993::Storage.TaskManager.Task::(_decref) Task=`34c31c95-f079-4bc9-9d64-f6cd6f2d8518`::ref 0 aborting False >Thread-70708::DEBUG::2014-09-25 17:36:34,975::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70708::DEBUG::2014-09-25 17:36:34,976::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:34,988::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70709::DEBUG::2014-09-25 17:36:34,990::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:36:34,990::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70709::DEBUG::2014-09-25 17:36:34,992::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5d8a3ba-92b9-460b-b23b-34ad90014e33`::moving from state init -> state preparing >Thread-70709::INFO::2014-09-25 17:36:34,992::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70709::DEBUG::2014-09-25 17:36:34,993::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`21a030db-1572-4a5e-8e96-59a056c2f595`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70709::DEBUG::2014-09-25 17:36:34,993::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70709::DEBUG::2014-09-25 17:36:34,993::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70709::DEBUG::2014-09-25 17:36:34,994::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`21a030db-1572-4a5e-8e96-59a056c2f595`::Granted request >Thread-70709::DEBUG::2014-09-25 17:36:34,994::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f5d8a3ba-92b9-460b-b23b-34ad90014e33`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70709::DEBUG::2014-09-25 17:36:34,994::task::993::Storage.TaskManager.Task::(_decref) Task=`f5d8a3ba-92b9-460b-b23b-34ad90014e33`::ref 1 aborting False >Thread-70709::INFO::2014-09-25 17:36:34,999::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70709::DEBUG::2014-09-25 17:36:34,999::task::1191::Storage.TaskManager.Task::(prepare) Task=`f5d8a3ba-92b9-460b-b23b-34ad90014e33`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70709::DEBUG::2014-09-25 17:36:34,999::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5d8a3ba-92b9-460b-b23b-34ad90014e33`::moving from state preparing -> state finished >Thread-70709::DEBUG::2014-09-25 17:36:35,000::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70709::DEBUG::2014-09-25 17:36:35,000::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70709::DEBUG::2014-09-25 17:36:35,000::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70709::DEBUG::2014-09-25 17:36:35,000::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70709::DEBUG::2014-09-25 17:36:35,001::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70709::DEBUG::2014-09-25 17:36:35,001::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70709::DEBUG::2014-09-25 17:36:35,001::task::993::Storage.TaskManager.Task::(_decref) Task=`f5d8a3ba-92b9-460b-b23b-34ad90014e33`::ref 0 aborting False >Thread-70709::DEBUG::2014-09-25 17:36:35,002::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70709::DEBUG::2014-09-25 17:36:35,002::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:36:35,051::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:35,051::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:35,052::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:35,052::__init__::232::IOProcess::(_processLogs) (108779) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:35,052::__init__::232::IOProcess::(_processLogs) (108779) Queuing response >Thread-80::DEBUG::2014-09-25 17:36:35,054::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:36:35,062::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000407817 s, 927 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:35,063::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:35,064::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:35,064::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:35,064::__init__::232::IOProcess::(_processLogs) (108780) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:35,065::__init__::232::IOProcess::(_processLogs) (108780) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:35,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:35,255::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164413 s, 62.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:35,591::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:35,592::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70710::DEBUG::2014-09-25 17:36:35,595::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:36:37,164::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:37,164::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:37,164::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:37,165::__init__::232::IOProcess::(_processLogs) (108781) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:37,171::__init__::232::IOProcess::(_processLogs) (108781) Queuing response >Thread-79::DEBUG::2014-09-25 17:36:37,173::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:36:37,188::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00483269 s, 116 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:37,190::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:37,190::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:37,190::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:37,191::__init__::232::IOProcess::(_processLogs) (108782) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:37,194::__init__::232::IOProcess::(_processLogs) (108782) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:37,195::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:37,195::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:37,196::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:37,196::__init__::232::IOProcess::(_processLogs) (108783) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:37,197::__init__::232::IOProcess::(_processLogs) (108783) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:37,197::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:37,197::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:37,198::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:37,198::__init__::232::IOProcess::(_processLogs) (108784) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:37,198::__init__::232::IOProcess::(_processLogs) (108784) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:37,199::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:37,199::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:37,200::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:37,200::__init__::232::IOProcess::(_processLogs) (108785) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:37,200::__init__::232::IOProcess::(_processLogs) (108785) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:37,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:37,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0364093 s, 28.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:38,611::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:38,612::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70711::DEBUG::2014-09-25 17:36:38,614::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:39,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:39,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016769 s, 61.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:36:41,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:41,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133613 s, 76.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:41,629::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:41,631::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70712::DEBUG::2014-09-25 17:36:41,633::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:36:41,826::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:41,827::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:41,827::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:41,827::__init__::232::IOProcess::(_processLogs) (108786) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:41,828::__init__::232::IOProcess::(_processLogs) (108786) Queuing response >Thread-81::DEBUG::2014-09-25 17:36:41,828::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:36:41,846::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00355767 s, 104 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:41,847::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:41,847::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:41,847::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:41,848::__init__::232::IOProcess::(_processLogs) (108787) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:41,851::__init__::232::IOProcess::(_processLogs) (108787) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:43,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:43,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203965 s, 50.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:44,645::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:44,647::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70713::DEBUG::2014-09-25 17:36:44,649::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:45,029::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70714::DEBUG::2014-09-25 17:36:45,030::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:36:45,030::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70714::DEBUG::2014-09-25 17:36:45,033::task::595::Storage.TaskManager.Task::(_updateState) Task=`e22a7c65-62ba-4a56-856a-2a8fa437ceb7`::moving from state init -> state preparing >Thread-70714::INFO::2014-09-25 17:36:45,033::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70714::INFO::2014-09-25 17:36:45,034::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70714::DEBUG::2014-09-25 17:36:45,034::task::1191::Storage.TaskManager.Task::(prepare) Task=`e22a7c65-62ba-4a56-856a-2a8fa437ceb7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70714::DEBUG::2014-09-25 17:36:45,034::task::595::Storage.TaskManager.Task::(_updateState) Task=`e22a7c65-62ba-4a56-856a-2a8fa437ceb7`::moving from state preparing -> state finished >Thread-70714::DEBUG::2014-09-25 17:36:45,035::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70714::DEBUG::2014-09-25 17:36:45,035::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70714::DEBUG::2014-09-25 17:36:45,035::task::993::Storage.TaskManager.Task::(_decref) Task=`e22a7c65-62ba-4a56-856a-2a8fa437ceb7`::ref 0 aborting False >Thread-70714::DEBUG::2014-09-25 17:36:45,035::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70714::DEBUG::2014-09-25 17:36:45,036::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:45,049::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70715::DEBUG::2014-09-25 17:36:45,050::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:36:45,050::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70715::DEBUG::2014-09-25 17:36:45,053::task::595::Storage.TaskManager.Task::(_updateState) Task=`0534ee1f-b1ca-463f-9b8a-434259eb9842`::moving from state init -> state preparing >Thread-70715::INFO::2014-09-25 17:36:45,053::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70715::DEBUG::2014-09-25 17:36:45,054::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`23736391-19d6-496a-8aad-a52df2e7e53e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70715::DEBUG::2014-09-25 17:36:45,054::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70715::DEBUG::2014-09-25 17:36:45,055::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70715::DEBUG::2014-09-25 17:36:45,055::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`23736391-19d6-496a-8aad-a52df2e7e53e`::Granted request >Thread-70715::DEBUG::2014-09-25 17:36:45,055::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0534ee1f-b1ca-463f-9b8a-434259eb9842`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70715::DEBUG::2014-09-25 17:36:45,056::task::993::Storage.TaskManager.Task::(_decref) Task=`0534ee1f-b1ca-463f-9b8a-434259eb9842`::ref 1 aborting False >Thread-70715::INFO::2014-09-25 17:36:45,061::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70715::DEBUG::2014-09-25 17:36:45,061::task::1191::Storage.TaskManager.Task::(prepare) Task=`0534ee1f-b1ca-463f-9b8a-434259eb9842`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70715::DEBUG::2014-09-25 17:36:45,061::task::595::Storage.TaskManager.Task::(_updateState) Task=`0534ee1f-b1ca-463f-9b8a-434259eb9842`::moving from state preparing -> state finished >Thread-70715::DEBUG::2014-09-25 17:36:45,061::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70715::DEBUG::2014-09-25 17:36:45,062::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70715::DEBUG::2014-09-25 17:36:45,062::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70715::DEBUG::2014-09-25 17:36:45,062::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70715::DEBUG::2014-09-25 17:36:45,063::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70715::DEBUG::2014-09-25 17:36:45,063::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70715::DEBUG::2014-09-25 17:36:45,063::task::993::Storage.TaskManager.Task::(_decref) Task=`0534ee1f-b1ca-463f-9b8a-434259eb9842`::ref 0 aborting False >Thread-70715::DEBUG::2014-09-25 17:36:45,064::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70715::DEBUG::2014-09-25 17:36:45,065::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:36:45,067::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:45,067::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:45,067::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:45,068::__init__::232::IOProcess::(_processLogs) (108788) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:45,068::__init__::232::IOProcess::(_processLogs) (108788) Queuing response >Thread-80::DEBUG::2014-09-25 17:36:45,069::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:36:45,079::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000415765 s, 909 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:45,080::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:45,080::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:45,081::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:45,081::__init__::232::IOProcess::(_processLogs) (108789) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:45,082::__init__::232::IOProcess::(_processLogs) (108789) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:45,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:45,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162843 s, 62.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:47,201::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:47,202::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:47,202::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:47,202::__init__::232::IOProcess::(_processLogs) (108790) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:47,202::__init__::232::IOProcess::(_processLogs) (108790) Queuing response >Thread-79::DEBUG::2014-09-25 17:36:47,203::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:36:47,210::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000382211 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:47,211::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:47,211::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:47,212::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:47,212::__init__::232::IOProcess::(_processLogs) (108791) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:47,212::__init__::232::IOProcess::(_processLogs) (108791) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:47,213::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:47,213::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:47,213::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:47,213::__init__::232::IOProcess::(_processLogs) (108792) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:47,213::__init__::232::IOProcess::(_processLogs) (108792) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:47,214::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:47,214::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:47,215::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:47,215::__init__::232::IOProcess::(_processLogs) (108793) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:47,215::__init__::232::IOProcess::(_processLogs) (108793) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:47,215::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:47,216::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:47,216::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:47,216::__init__::232::IOProcess::(_processLogs) (108794) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:47,216::__init__::232::IOProcess::(_processLogs) (108794) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:47,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:47,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020679 s, 49.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:47,661::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:47,663::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70716::DEBUG::2014-09-25 17:36:47,665::task::595::Storage.TaskManager.Task::(_updateState) Task=`51ac2d4d-a4ea-4634-bdb6-814c0f2a897b`::moving from state init -> state preparing >Thread-70716::INFO::2014-09-25 17:36:47,666::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70716::INFO::2014-09-25 17:36:47,666::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000382211', 'lastCheck': '0.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000415765', 'lastCheck': '2.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00355767', 'lastCheck': '5.8', 'valid': True}} >Thread-70716::DEBUG::2014-09-25 17:36:47,666::task::1191::Storage.TaskManager.Task::(prepare) Task=`51ac2d4d-a4ea-4634-bdb6-814c0f2a897b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000382211', 'lastCheck': '0.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000415765', 'lastCheck': '2.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00355767', 'lastCheck': '5.8', 'valid': True}} >Thread-70716::DEBUG::2014-09-25 17:36:47,667::task::595::Storage.TaskManager.Task::(_updateState) Task=`51ac2d4d-a4ea-4634-bdb6-814c0f2a897b`::moving from state preparing -> state finished >Thread-70716::DEBUG::2014-09-25 17:36:47,667::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70716::DEBUG::2014-09-25 17:36:47,667::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70716::DEBUG::2014-09-25 17:36:47,667::task::993::Storage.TaskManager.Task::(_decref) Task=`51ac2d4d-a4ea-4634-bdb6-814c0f2a897b`::ref 0 aborting False >Thread-70716::DEBUG::2014-09-25 17:36:47,670::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:47,686::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:47,688::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70717::DEBUG::2014-09-25 17:36:47,689::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:49,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:49,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0190016 s, 53.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:50,735::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:50,737::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70718::DEBUG::2014-09-25 17:36:50,739::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:51,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:51,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200176 s, 51.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:51,854::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:51,855::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:51,856::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:51,857::__init__::232::IOProcess::(_processLogs) (108795) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:51,862::__init__::232::IOProcess::(_processLogs) (108795) Queuing response >Thread-81::DEBUG::2014-09-25 17:36:51,862::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:36:51,879::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00346222 s, 107 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:51,880::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:51,880::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:51,880::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:51,881::__init__::232::IOProcess::(_processLogs) (108796) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:51,886::__init__::232::IOProcess::(_processLogs) (108796) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:53,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:53,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153906 s, 66.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:53,751::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:53,754::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70719::DEBUG::2014-09-25 17:36:53,762::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:36:55,084::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:55,084::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:55,085::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:55,085::__init__::232::IOProcess::(_processLogs) (108797) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:55,086::__init__::232::IOProcess::(_processLogs) (108797) Queuing response >Thread-80::DEBUG::2014-09-25 17:36:55,086::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:55,095::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-80::DEBUG::2014-09-25 17:36:55,097::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000469697 s, 805 kB/s\n'; <rc> = 0 >JsonRpcServer::DEBUG::2014-09-25 17:36:55,098::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70720::DEBUG::2014-09-25 17:36:55,099::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-29::DEBUG::2014-09-25 17:36:55,102::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:55,103::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:55,104::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-70720::DEBUG::2014-09-25 17:36:55,105::task::595::Storage.TaskManager.Task::(_updateState) Task=`35e88828-df0d-4303-b4c6-1c44a4ee419d`::moving from state init -> state preparing >Thread-29::DEBUG::2014-09-25 17:36:55,106::__init__::232::IOProcess::(_processLogs) (108798) Got request for method 'statvfs' >Thread-70720::INFO::2014-09-25 17:36:55,107::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-29::DEBUG::2014-09-25 17:36:55,108::__init__::232::IOProcess::(_processLogs) (108798) Queuing response >Thread-70720::INFO::2014-09-25 17:36:55,110::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70720::DEBUG::2014-09-25 17:36:55,111::task::1191::Storage.TaskManager.Task::(prepare) Task=`35e88828-df0d-4303-b4c6-1c44a4ee419d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70720::DEBUG::2014-09-25 17:36:55,111::task::595::Storage.TaskManager.Task::(_updateState) Task=`35e88828-df0d-4303-b4c6-1c44a4ee419d`::moving from state preparing -> state finished >Thread-70720::DEBUG::2014-09-25 17:36:55,111::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70720::DEBUG::2014-09-25 17:36:55,111::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70720::DEBUG::2014-09-25 17:36:55,112::task::993::Storage.TaskManager.Task::(_decref) Task=`35e88828-df0d-4303-b4c6-1c44a4ee419d`::ref 0 aborting False >Thread-70720::DEBUG::2014-09-25 17:36:55,112::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70720::DEBUG::2014-09-25 17:36:55,112::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:55,123::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70721::DEBUG::2014-09-25 17:36:55,124::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:36:55,124::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70721::DEBUG::2014-09-25 17:36:55,126::task::595::Storage.TaskManager.Task::(_updateState) Task=`7c292021-fb70-4094-a03a-cf5efd75fd8b`::moving from state init -> state preparing >Thread-70721::INFO::2014-09-25 17:36:55,127::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70721::DEBUG::2014-09-25 17:36:55,127::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`29b28bc3-6fad-4632-8434-91ac859a38b4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70721::DEBUG::2014-09-25 17:36:55,128::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70721::DEBUG::2014-09-25 17:36:55,128::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70721::DEBUG::2014-09-25 17:36:55,128::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`29b28bc3-6fad-4632-8434-91ac859a38b4`::Granted request >Thread-70721::DEBUG::2014-09-25 17:36:55,129::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7c292021-fb70-4094-a03a-cf5efd75fd8b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70721::DEBUG::2014-09-25 17:36:55,129::task::993::Storage.TaskManager.Task::(_decref) Task=`7c292021-fb70-4094-a03a-cf5efd75fd8b`::ref 1 aborting False >Thread-70721::INFO::2014-09-25 17:36:55,133::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70721::DEBUG::2014-09-25 17:36:55,134::task::1191::Storage.TaskManager.Task::(prepare) Task=`7c292021-fb70-4094-a03a-cf5efd75fd8b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70721::DEBUG::2014-09-25 17:36:55,134::task::595::Storage.TaskManager.Task::(_updateState) Task=`7c292021-fb70-4094-a03a-cf5efd75fd8b`::moving from state preparing -> state finished >Thread-70721::DEBUG::2014-09-25 17:36:55,134::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70721::DEBUG::2014-09-25 17:36:55,134::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70721::DEBUG::2014-09-25 17:36:55,135::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70721::DEBUG::2014-09-25 17:36:55,135::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70721::DEBUG::2014-09-25 17:36:55,135::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70721::DEBUG::2014-09-25 17:36:55,135::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70721::DEBUG::2014-09-25 17:36:55,136::task::993::Storage.TaskManager.Task::(_decref) Task=`7c292021-fb70-4094-a03a-cf5efd75fd8b`::ref 0 aborting False >Thread-70721::DEBUG::2014-09-25 17:36:55,136::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70721::DEBUG::2014-09-25 17:36:55,137::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:55,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:55,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199028 s, 51.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:56,778::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:56,779::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70722::DEBUG::2014-09-25 17:36:56,781::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:36:57,218::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:57,218::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:57,218::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:57,218::__init__::232::IOProcess::(_processLogs) (108799) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:57,226::__init__::232::IOProcess::(_processLogs) (108799) Queuing response >Thread-79::DEBUG::2014-09-25 17:36:57,227::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:36:57,260::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0102651 s, 54.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:36:57,261::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:57,261::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:57,262::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:57,262::__init__::232::IOProcess::(_processLogs) (108800) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:36:57,270::__init__::232::IOProcess::(_processLogs) (108800) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:57,271::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:57,271::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:57,272::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:57,273::__init__::232::IOProcess::(_processLogs) (108801) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:57,273::__init__::232::IOProcess::(_processLogs) (108801) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:57,273::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:57,274::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:57,274::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:57,275::__init__::232::IOProcess::(_processLogs) (108802) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:57,275::__init__::232::IOProcess::(_processLogs) (108802) Queuing response >Thread-29::DEBUG::2014-09-25 17:36:57,275::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:36:57,276::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:36:57,276::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:36:57,277::__init__::232::IOProcess::(_processLogs) (108803) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:36:57,277::__init__::232::IOProcess::(_processLogs) (108803) Queuing response >Dummy-148::DEBUG::2014-09-25 17:36:57,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:57,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135147 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:36:59,795::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:36:59,796::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70723::DEBUG::2014-09-25 17:36:59,798::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:36:59,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:36:59,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.02232 s, 45.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:01,888::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:01,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:01,888::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:01,889::__init__::232::IOProcess::(_processLogs) (108804) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:01,889::__init__::232::IOProcess::(_processLogs) (108804) Queuing response >Thread-81::DEBUG::2014-09-25 17:37:01,890::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:37:01,900::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000359946 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:01,904::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:01,905::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:01,906::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:01,907::__init__::232::IOProcess::(_processLogs) (108805) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:01,909::__init__::232::IOProcess::(_processLogs) (108805) Queuing response >Dummy-148::DEBUG::2014-09-25 17:37:01,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:01,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163335 s, 62.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:02,811::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:02,812::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70724::DEBUG::2014-09-25 17:37:02,815::task::595::Storage.TaskManager.Task::(_updateState) Task=`52302756-8cc9-472f-b5c6-3d9ee2cdd0b7`::moving from state init -> state preparing >Thread-70724::INFO::2014-09-25 17:37:02,815::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70724::INFO::2014-09-25 17:37:02,816::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0102651', 'lastCheck': '5.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000469697', 'lastCheck': '7.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000359946', 'lastCheck': '0.9', 'valid': True}} >Thread-70724::DEBUG::2014-09-25 17:37:02,816::task::1191::Storage.TaskManager.Task::(prepare) Task=`52302756-8cc9-472f-b5c6-3d9ee2cdd0b7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0102651', 'lastCheck': '5.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000469697', 'lastCheck': '7.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000359946', 'lastCheck': '0.9', 'valid': True}} >Thread-70724::DEBUG::2014-09-25 17:37:02,816::task::595::Storage.TaskManager.Task::(_updateState) Task=`52302756-8cc9-472f-b5c6-3d9ee2cdd0b7`::moving from state preparing -> state finished >Thread-70724::DEBUG::2014-09-25 17:37:02,816::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70724::DEBUG::2014-09-25 17:37:02,817::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70724::DEBUG::2014-09-25 17:37:02,817::task::993::Storage.TaskManager.Task::(_decref) Task=`52302756-8cc9-472f-b5c6-3d9ee2cdd0b7`::ref 0 aborting False >Thread-70724::DEBUG::2014-09-25 17:37:02,819::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:02,839::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:02,840::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70725::DEBUG::2014-09-25 17:37:02,844::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:03,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:03,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147608 s, 69.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:05,111::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:05,112::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:05,112::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:37:05,114::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:37:05,114::__init__::232::IOProcess::(_processLogs) (108806) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:05,122::__init__::232::IOProcess::(_processLogs) (108806) Queuing response >Thread-80::DEBUG::2014-09-25 17:37:05,123::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00038867 s, 973 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:05,124::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:05,124::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:05,125::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:05,125::__init__::232::IOProcess::(_processLogs) (108807) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:05,126::__init__::232::IOProcess::(_processLogs) (108807) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:05,160::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70726::DEBUG::2014-09-25 17:37:05,161::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:37:05,161::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70726::DEBUG::2014-09-25 17:37:05,164::task::595::Storage.TaskManager.Task::(_updateState) Task=`28500b8a-4e0f-43d4-b48a-adf60ce8d3c8`::moving from state init -> state preparing >Thread-70726::INFO::2014-09-25 17:37:05,164::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70726::INFO::2014-09-25 17:37:05,165::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70726::DEBUG::2014-09-25 17:37:05,165::task::1191::Storage.TaskManager.Task::(prepare) Task=`28500b8a-4e0f-43d4-b48a-adf60ce8d3c8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70726::DEBUG::2014-09-25 17:37:05,165::task::595::Storage.TaskManager.Task::(_updateState) Task=`28500b8a-4e0f-43d4-b48a-adf60ce8d3c8`::moving from state preparing -> state finished >Thread-70726::DEBUG::2014-09-25 17:37:05,165::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70726::DEBUG::2014-09-25 17:37:05,165::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70726::DEBUG::2014-09-25 17:37:05,166::task::993::Storage.TaskManager.Task::(_decref) Task=`28500b8a-4e0f-43d4-b48a-adf60ce8d3c8`::ref 0 aborting False >Thread-70726::DEBUG::2014-09-25 17:37:05,166::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70726::DEBUG::2014-09-25 17:37:05,166::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:05,177::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70727::DEBUG::2014-09-25 17:37:05,178::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:37:05,178::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70727::DEBUG::2014-09-25 17:37:05,180::task::595::Storage.TaskManager.Task::(_updateState) Task=`cf55d694-054b-4021-a78f-acf6e148e7cd`::moving from state init -> state preparing >Thread-70727::INFO::2014-09-25 17:37:05,181::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70727::DEBUG::2014-09-25 17:37:05,181::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ea585fab-1704-4675-b298-79027d6ed9ac`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70727::DEBUG::2014-09-25 17:37:05,182::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70727::DEBUG::2014-09-25 17:37:05,182::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70727::DEBUG::2014-09-25 17:37:05,182::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ea585fab-1704-4675-b298-79027d6ed9ac`::Granted request >Thread-70727::DEBUG::2014-09-25 17:37:05,183::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`cf55d694-054b-4021-a78f-acf6e148e7cd`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70727::DEBUG::2014-09-25 17:37:05,183::task::993::Storage.TaskManager.Task::(_decref) Task=`cf55d694-054b-4021-a78f-acf6e148e7cd`::ref 1 aborting False >Thread-70727::INFO::2014-09-25 17:37:05,187::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70727::DEBUG::2014-09-25 17:37:05,188::task::1191::Storage.TaskManager.Task::(prepare) Task=`cf55d694-054b-4021-a78f-acf6e148e7cd`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70727::DEBUG::2014-09-25 17:37:05,188::task::595::Storage.TaskManager.Task::(_updateState) Task=`cf55d694-054b-4021-a78f-acf6e148e7cd`::moving from state preparing -> state finished >Thread-70727::DEBUG::2014-09-25 17:37:05,188::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70727::DEBUG::2014-09-25 17:37:05,188::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70727::DEBUG::2014-09-25 17:37:05,189::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70727::DEBUG::2014-09-25 17:37:05,189::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70727::DEBUG::2014-09-25 17:37:05,189::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70727::DEBUG::2014-09-25 17:37:05,190::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70727::DEBUG::2014-09-25 17:37:05,190::task::993::Storage.TaskManager.Task::(_decref) Task=`cf55d694-054b-4021-a78f-acf6e148e7cd`::ref 0 aborting False >Thread-70727::DEBUG::2014-09-25 17:37:05,190::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70727::DEBUG::2014-09-25 17:37:05,191::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:05,899::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:05,901::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70728::DEBUG::2014-09-25 17:37:05,902::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:06,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:06,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0140534 s, 72.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:07,278::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:07,279::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:07,279::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:07,279::__init__::232::IOProcess::(_processLogs) (108808) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:07,283::__init__::232::IOProcess::(_processLogs) (108808) Queuing response >Thread-79::DEBUG::2014-09-25 17:37:07,285::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:37:07,309::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00504749 s, 111 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:07,311::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:07,311::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:07,311::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:07,311::__init__::232::IOProcess::(_processLogs) (108809) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:07,316::__init__::232::IOProcess::(_processLogs) (108809) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:07,318::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:07,318::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:07,319::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:07,319::__init__::232::IOProcess::(_processLogs) (108810) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:07,319::__init__::232::IOProcess::(_processLogs) (108810) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:07,320::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:07,320::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:07,321::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:07,321::__init__::232::IOProcess::(_processLogs) (108811) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:07,321::__init__::232::IOProcess::(_processLogs) (108811) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:07,322::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:07,322::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:07,323::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:07,323::__init__::232::IOProcess::(_processLogs) (108812) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:07,324::__init__::232::IOProcess::(_processLogs) (108812) Queuing response >Dummy-148::DEBUG::2014-09-25 17:37:08,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:08,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156537 s, 65.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:08,917::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:08,918::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70729::DEBUG::2014-09-25 17:37:08,920::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:10,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:10,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0272169 s, 37.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:11,912::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:11,913::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:11,913::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:11,913::__init__::232::IOProcess::(_processLogs) (108813) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:11,919::__init__::232::IOProcess::(_processLogs) (108813) Queuing response >Thread-81::DEBUG::2014-09-25 17:37:11,920::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:11,933::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:11,935::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70730::DEBUG::2014-09-25 17:37:11,937::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 17:37:11,938::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.004689 s, 79.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:11,939::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:11,939::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:11,940::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:11,940::__init__::232::IOProcess::(_processLogs) (108814) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:11,942::__init__::232::IOProcess::(_processLogs) (108814) Queuing response >Dummy-148::DEBUG::2014-09-25 17:37:12,179::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:12,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183128 s, 55.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:37:14,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:14,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172971 s, 59.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:14,948::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:14,950::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70731::DEBUG::2014-09-25 17:37:14,952::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:37:15,128::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:15,128::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:15,128::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:15,128::__init__::232::IOProcess::(_processLogs) (108815) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:15,129::__init__::232::IOProcess::(_processLogs) (108815) Queuing response >Thread-80::DEBUG::2014-09-25 17:37:15,130::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:37:15,139::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000371599 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:15,140::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:15,140::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:15,141::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:15,142::__init__::232::IOProcess::(_processLogs) (108816) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:15,143::__init__::232::IOProcess::(_processLogs) (108816) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:15,214::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70732::DEBUG::2014-09-25 17:37:15,216::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:37:15,216::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70732::DEBUG::2014-09-25 17:37:15,218::task::595::Storage.TaskManager.Task::(_updateState) Task=`37708969-468e-4af7-95fe-128af55d9b65`::moving from state init -> state preparing >Thread-70732::INFO::2014-09-25 17:37:15,219::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70732::INFO::2014-09-25 17:37:15,219::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70732::DEBUG::2014-09-25 17:37:15,220::task::1191::Storage.TaskManager.Task::(prepare) Task=`37708969-468e-4af7-95fe-128af55d9b65`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70732::DEBUG::2014-09-25 17:37:15,220::task::595::Storage.TaskManager.Task::(_updateState) Task=`37708969-468e-4af7-95fe-128af55d9b65`::moving from state preparing -> state finished >Thread-70732::DEBUG::2014-09-25 17:37:15,220::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70732::DEBUG::2014-09-25 17:37:15,220::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70732::DEBUG::2014-09-25 17:37:15,220::task::993::Storage.TaskManager.Task::(_decref) Task=`37708969-468e-4af7-95fe-128af55d9b65`::ref 0 aborting False >Thread-70732::DEBUG::2014-09-25 17:37:15,221::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70732::DEBUG::2014-09-25 17:37:15,221::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:15,232::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:15,233::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70733::DEBUG::2014-09-25 17:37:15,233::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70733::DEBUG::2014-09-25 17:37:15,235::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc70d829-ab94-4ef8-9d0c-86b4508498b7`::moving from state init -> state preparing >Thread-70733::INFO::2014-09-25 17:37:15,235::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70733::DEBUG::2014-09-25 17:37:15,236::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f5efac28-255b-4621-baec-6a0145d97559`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70733::DEBUG::2014-09-25 17:37:15,236::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70733::DEBUG::2014-09-25 17:37:15,236::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70733::DEBUG::2014-09-25 17:37:15,237::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f5efac28-255b-4621-baec-6a0145d97559`::Granted request >Thread-70733::DEBUG::2014-09-25 17:37:15,238::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`dc70d829-ab94-4ef8-9d0c-86b4508498b7`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70733::DEBUG::2014-09-25 17:37:15,239::task::993::Storage.TaskManager.Task::(_decref) Task=`dc70d829-ab94-4ef8-9d0c-86b4508498b7`::ref 1 aborting False >Thread-70733::INFO::2014-09-25 17:37:15,243::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70733::DEBUG::2014-09-25 17:37:15,243::task::1191::Storage.TaskManager.Task::(prepare) Task=`dc70d829-ab94-4ef8-9d0c-86b4508498b7`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70733::DEBUG::2014-09-25 17:37:15,244::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc70d829-ab94-4ef8-9d0c-86b4508498b7`::moving from state preparing -> state finished >Thread-70733::DEBUG::2014-09-25 17:37:15,244::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70733::DEBUG::2014-09-25 17:37:15,244::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70733::DEBUG::2014-09-25 17:37:15,244::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70733::DEBUG::2014-09-25 17:37:15,245::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70733::DEBUG::2014-09-25 17:37:15,245::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70733::DEBUG::2014-09-25 17:37:15,245::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70733::DEBUG::2014-09-25 17:37:15,246::task::993::Storage.TaskManager.Task::(_decref) Task=`dc70d829-ab94-4ef8-9d0c-86b4508498b7`::ref 0 aborting False >Thread-70733::DEBUG::2014-09-25 17:37:15,246::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70733::DEBUG::2014-09-25 17:37:15,247::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:16,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:16,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0249057 s, 41.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:17,324::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:17,324::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:17,325::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:17,325::__init__::232::IOProcess::(_processLogs) (108817) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:17,325::__init__::232::IOProcess::(_processLogs) (108817) Queuing response >Thread-79::DEBUG::2014-09-25 17:37:17,327::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:37:17,335::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000447939 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:17,337::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:17,337::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:17,338::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:17,339::__init__::232::IOProcess::(_processLogs) (108818) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:17,339::__init__::232::IOProcess::(_processLogs) (108818) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:17,339::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:17,340::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:17,340::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:17,341::__init__::232::IOProcess::(_processLogs) (108819) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:17,341::__init__::232::IOProcess::(_processLogs) (108819) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:17,342::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:17,342::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:17,342::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:17,343::__init__::232::IOProcess::(_processLogs) (108820) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:17,343::__init__::232::IOProcess::(_processLogs) (108820) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:17,344::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:17,344::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:17,344::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:17,345::__init__::232::IOProcess::(_processLogs) (108821) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:17,345::__init__::232::IOProcess::(_processLogs) (108821) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:17,964::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:17,965::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70734::DEBUG::2014-09-25 17:37:17,973::task::595::Storage.TaskManager.Task::(_updateState) Task=`9b190b83-a185-447b-961c-369ef7d623d2`::moving from state init -> state preparing >Thread-70734::INFO::2014-09-25 17:37:17,974::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70734::INFO::2014-09-25 17:37:17,976::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000447939', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000371599', 'lastCheck': '2.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.004689', 'lastCheck': '6.0', 'valid': True}} >Thread-70734::DEBUG::2014-09-25 17:37:17,977::task::1191::Storage.TaskManager.Task::(prepare) Task=`9b190b83-a185-447b-961c-369ef7d623d2`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000447939', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000371599', 'lastCheck': '2.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.004689', 'lastCheck': '6.0', 'valid': True}} >Thread-70734::DEBUG::2014-09-25 17:37:17,978::task::595::Storage.TaskManager.Task::(_updateState) Task=`9b190b83-a185-447b-961c-369ef7d623d2`::moving from state preparing -> state finished >Thread-70734::DEBUG::2014-09-25 17:37:17,978::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70734::DEBUG::2014-09-25 17:37:17,979::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70734::DEBUG::2014-09-25 17:37:17,980::task::993::Storage.TaskManager.Task::(_decref) Task=`9b190b83-a185-447b-961c-369ef7d623d2`::ref 0 aborting False >Thread-70734::DEBUG::2014-09-25 17:37:17,987::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:18,005::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:18,006::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70735::DEBUG::2014-09-25 17:37:18,008::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:18,509::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:18,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168768 s, 60.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:37:20,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:20,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209837 s, 48.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:21,052::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:21,053::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70736::DEBUG::2014-09-25 17:37:21,055::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:37:21,944::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:21,945::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:21,945::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:21,945::__init__::232::IOProcess::(_processLogs) (108822) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:21,946::__init__::232::IOProcess::(_processLogs) (108822) Queuing response >Thread-81::DEBUG::2014-09-25 17:37:21,947::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:37:21,964::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00420764 s, 88.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:21,966::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:21,966::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:21,966::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:21,966::__init__::232::IOProcess::(_processLogs) (108823) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:21,968::__init__::232::IOProcess::(_processLogs) (108823) Queuing response >Dummy-148::DEBUG::2014-09-25 17:37:22,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:22,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148319 s, 69.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:24,069::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:24,071::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70737::DEBUG::2014-09-25 17:37:24,073::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:24,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:24,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135164 s, 75.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:25,143::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:25,144::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:25,144::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:25,144::__init__::232::IOProcess::(_processLogs) (108824) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:25,145::__init__::232::IOProcess::(_processLogs) (108824) Queuing response >Thread-80::DEBUG::2014-09-25 17:37:25,146::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:37:25,155::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000714447 s, 529 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:25,157::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:25,157::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:25,157::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:25,158::__init__::232::IOProcess::(_processLogs) (108825) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:25,158::__init__::232::IOProcess::(_processLogs) (108825) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:25,275::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70738::DEBUG::2014-09-25 17:37:25,276::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:37:25,276::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70738::DEBUG::2014-09-25 17:37:25,279::task::595::Storage.TaskManager.Task::(_updateState) Task=`d8e4af34-f8de-4a7c-96e9-06dc4f536724`::moving from state init -> state preparing >Thread-70738::INFO::2014-09-25 17:37:25,279::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70738::INFO::2014-09-25 17:37:25,280::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70738::DEBUG::2014-09-25 17:37:25,280::task::1191::Storage.TaskManager.Task::(prepare) Task=`d8e4af34-f8de-4a7c-96e9-06dc4f536724`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70738::DEBUG::2014-09-25 17:37:25,280::task::595::Storage.TaskManager.Task::(_updateState) Task=`d8e4af34-f8de-4a7c-96e9-06dc4f536724`::moving from state preparing -> state finished >Thread-70738::DEBUG::2014-09-25 17:37:25,280::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70738::DEBUG::2014-09-25 17:37:25,280::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70738::DEBUG::2014-09-25 17:37:25,281::task::993::Storage.TaskManager.Task::(_decref) Task=`d8e4af34-f8de-4a7c-96e9-06dc4f536724`::ref 0 aborting False >Thread-70738::DEBUG::2014-09-25 17:37:25,281::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70738::DEBUG::2014-09-25 17:37:25,281::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:25,303::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70739::DEBUG::2014-09-25 17:37:25,304::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:37:25,304::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70739::DEBUG::2014-09-25 17:37:25,306::task::595::Storage.TaskManager.Task::(_updateState) Task=`029c0c87-92f0-472c-bb5e-2f6b120c73a8`::moving from state init -> state preparing >Thread-70739::INFO::2014-09-25 17:37:25,306::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70739::DEBUG::2014-09-25 17:37:25,307::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9e907752-9335-4506-ba64-dfe355844de3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70739::DEBUG::2014-09-25 17:37:25,307::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70739::DEBUG::2014-09-25 17:37:25,308::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70739::DEBUG::2014-09-25 17:37:25,308::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9e907752-9335-4506-ba64-dfe355844de3`::Granted request >Thread-70739::DEBUG::2014-09-25 17:37:25,308::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`029c0c87-92f0-472c-bb5e-2f6b120c73a8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70739::DEBUG::2014-09-25 17:37:25,309::task::993::Storage.TaskManager.Task::(_decref) Task=`029c0c87-92f0-472c-bb5e-2f6b120c73a8`::ref 1 aborting False >Thread-70739::INFO::2014-09-25 17:37:25,313::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70739::DEBUG::2014-09-25 17:37:25,313::task::1191::Storage.TaskManager.Task::(prepare) Task=`029c0c87-92f0-472c-bb5e-2f6b120c73a8`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70739::DEBUG::2014-09-25 17:37:25,314::task::595::Storage.TaskManager.Task::(_updateState) Task=`029c0c87-92f0-472c-bb5e-2f6b120c73a8`::moving from state preparing -> state finished >Thread-70739::DEBUG::2014-09-25 17:37:25,314::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70739::DEBUG::2014-09-25 17:37:25,314::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70739::DEBUG::2014-09-25 17:37:25,314::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70739::DEBUG::2014-09-25 17:37:25,315::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70739::DEBUG::2014-09-25 17:37:25,315::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70739::DEBUG::2014-09-25 17:37:25,315::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70739::DEBUG::2014-09-25 17:37:25,315::task::993::Storage.TaskManager.Task::(_decref) Task=`029c0c87-92f0-472c-bb5e-2f6b120c73a8`::ref 0 aborting False >Thread-70739::DEBUG::2014-09-25 17:37:25,316::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70739::DEBUG::2014-09-25 17:37:25,316::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:26,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:26,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210695 s, 48.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:27,086::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:27,087::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70740::DEBUG::2014-09-25 17:37:27,089::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:37:27,346::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:27,346::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:27,347::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:27,347::__init__::232::IOProcess::(_processLogs) (108826) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:27,353::__init__::232::IOProcess::(_processLogs) (108826) Queuing response >Thread-79::DEBUG::2014-09-25 17:37:27,354::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:37:27,378::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00840759 s, 66.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:27,380::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:27,380::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:27,380::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:27,381::__init__::232::IOProcess::(_processLogs) (108827) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:27,387::__init__::232::IOProcess::(_processLogs) (108827) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:27,389::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:27,390::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:27,390::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:27,390::__init__::232::IOProcess::(_processLogs) (108828) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:27,391::__init__::232::IOProcess::(_processLogs) (108828) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:27,391::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:27,392::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:27,392::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:27,392::__init__::232::IOProcess::(_processLogs) (108829) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:27,393::__init__::232::IOProcess::(_processLogs) (108829) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:27,393::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:27,393::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:27,394::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:27,394::__init__::232::IOProcess::(_processLogs) (108830) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:27,394::__init__::232::IOProcess::(_processLogs) (108830) Queuing response >Dummy-148::DEBUG::2014-09-25 17:37:28,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:28,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169145 s, 60.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:30,104::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:30,105::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70741::DEBUG::2014-09-25 17:37:30,107::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:30,806::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:30,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163068 s, 62.8 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 17:37:31,969::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:37:31,974::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:31,974::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:31,974::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:31,975::__init__::232::IOProcess::(_processLogs) (108831) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:37:31,975::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:31,976::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:31,976::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:31,977::__init__::232::IOProcess::(_processLogs) (108832) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:37:31,977::__init__::232::IOProcess::(_processLogs) (108832) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:31,977::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:31,978::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:31,978::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:31,978::__init__::232::IOProcess::(_processLogs) (108833) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:37:31,984::__init__::232::IOProcess::(_processLogs) (108831) Queuing response >Thread-81::DEBUG::2014-09-25 17:37:31,988::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:37:31,989::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:31,989::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:31,989::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:31,990::__init__::232::IOProcess::(_processLogs) (108834) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:37:31,991::__init__::232::IOProcess::(_processLogs) (108833) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:32,022::__init__::232::IOProcess::(_processLogs) (108834) Queuing response >Thread-81::DEBUG::2014-09-25 17:37:32,022::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:37:32,024::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:32,024::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:32,024::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:32,024::__init__::232::IOProcess::(_processLogs) (108835) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:37:32,047::__init__::232::IOProcess::(_processLogs) (108835) Queuing response >Thread-81::DEBUG::2014-09-25 17:37:32,047::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:37:32,048::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:32,049::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:32,049::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:32,050::__init__::232::IOProcess::(_processLogs) (108836) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:32,050::__init__::232::IOProcess::(_processLogs) (108836) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:32,050::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:32,051::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:32,051::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:32,051::__init__::232::IOProcess::(_processLogs) (108837) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:37:32,059::__init__::232::IOProcess::(_processLogs) (108837) Queuing response >Thread-81::DEBUG::2014-09-25 17:37:32,059::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 17:37:32,059::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:37:32,059::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:37:32,060::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:32,060::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:32,061::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:32,061::__init__::232::IOProcess::(_processLogs) (108838) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:32,064::__init__::232::IOProcess::(_processLogs) (108838) Queuing response >Thread-81::DEBUG::2014-09-25 17:37:32,065::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:37:32,081::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00681327 s, 54.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:32,082::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:32,083::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:32,083::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:32,083::__init__::232::IOProcess::(_processLogs) (108839) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:32,096::__init__::232::IOProcess::(_processLogs) (108839) Queuing response >Dummy-148::DEBUG::2014-09-25 17:37:32,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:32,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0208935 s, 49.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:33,118::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:33,119::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70745::DEBUG::2014-09-25 17:37:33,122::task::595::Storage.TaskManager.Task::(_updateState) Task=`46af1e6d-1cdd-4444-88c1-42ffecfef5e3`::moving from state init -> state preparing >Thread-70745::INFO::2014-09-25 17:37:33,122::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70745::INFO::2014-09-25 17:37:33,123::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00840759', 'lastCheck': '5.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000714447', 'lastCheck': '8.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00681327', 'lastCheck': '1.0', 'valid': True}} >Thread-70745::DEBUG::2014-09-25 17:37:33,123::task::1191::Storage.TaskManager.Task::(prepare) Task=`46af1e6d-1cdd-4444-88c1-42ffecfef5e3`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00840759', 'lastCheck': '5.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000714447', 'lastCheck': '8.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00681327', 'lastCheck': '1.0', 'valid': True}} >Thread-70745::DEBUG::2014-09-25 17:37:33,123::task::595::Storage.TaskManager.Task::(_updateState) Task=`46af1e6d-1cdd-4444-88c1-42ffecfef5e3`::moving from state preparing -> state finished >Thread-70745::DEBUG::2014-09-25 17:37:33,123::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70745::DEBUG::2014-09-25 17:37:33,124::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70745::DEBUG::2014-09-25 17:37:33,124::task::993::Storage.TaskManager.Task::(_decref) Task=`46af1e6d-1cdd-4444-88c1-42ffecfef5e3`::ref 0 aborting False >Thread-70745::DEBUG::2014-09-25 17:37:33,126::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:33,139::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:33,141::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70746::DEBUG::2014-09-25 17:37:33,142::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:34,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:34,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156391 s, 65.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:35,160::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:35,161::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:35,161::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:35,161::__init__::232::IOProcess::(_processLogs) (108840) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:35,162::__init__::232::IOProcess::(_processLogs) (108840) Queuing response >Thread-80::DEBUG::2014-09-25 17:37:35,163::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:37:35,171::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000417875 s, 905 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:35,173::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:35,173::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:35,173::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:35,174::__init__::232::IOProcess::(_processLogs) (108841) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:35,174::__init__::232::IOProcess::(_processLogs) (108841) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:35,347::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70747::DEBUG::2014-09-25 17:37:35,348::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:37:35,348::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70747::DEBUG::2014-09-25 17:37:35,350::task::595::Storage.TaskManager.Task::(_updateState) Task=`01d32080-cb58-4d56-8dd3-25e8084fb103`::moving from state init -> state preparing >Thread-70747::INFO::2014-09-25 17:37:35,351::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70747::INFO::2014-09-25 17:37:35,351::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70747::DEBUG::2014-09-25 17:37:35,351::task::1191::Storage.TaskManager.Task::(prepare) Task=`01d32080-cb58-4d56-8dd3-25e8084fb103`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70747::DEBUG::2014-09-25 17:37:35,352::task::595::Storage.TaskManager.Task::(_updateState) Task=`01d32080-cb58-4d56-8dd3-25e8084fb103`::moving from state preparing -> state finished >Thread-70747::DEBUG::2014-09-25 17:37:35,352::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70747::DEBUG::2014-09-25 17:37:35,352::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70747::DEBUG::2014-09-25 17:37:35,352::task::993::Storage.TaskManager.Task::(_decref) Task=`01d32080-cb58-4d56-8dd3-25e8084fb103`::ref 0 aborting False >Thread-70747::DEBUG::2014-09-25 17:37:35,353::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70747::DEBUG::2014-09-25 17:37:35,353::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:35,366::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:35,367::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70748::DEBUG::2014-09-25 17:37:35,367::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70748::DEBUG::2014-09-25 17:37:35,369::task::595::Storage.TaskManager.Task::(_updateState) Task=`507459a6-f186-4426-9ef7-d816329519de`::moving from state init -> state preparing >Thread-70748::INFO::2014-09-25 17:37:35,369::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70748::DEBUG::2014-09-25 17:37:35,370::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`74d48f03-e45b-4291-8331-dab493f20439`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70748::DEBUG::2014-09-25 17:37:35,370::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70748::DEBUG::2014-09-25 17:37:35,371::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70748::DEBUG::2014-09-25 17:37:35,371::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`74d48f03-e45b-4291-8331-dab493f20439`::Granted request >Thread-70748::DEBUG::2014-09-25 17:37:35,372::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`507459a6-f186-4426-9ef7-d816329519de`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70748::DEBUG::2014-09-25 17:37:35,372::task::993::Storage.TaskManager.Task::(_decref) Task=`507459a6-f186-4426-9ef7-d816329519de`::ref 1 aborting False >Thread-70748::INFO::2014-09-25 17:37:35,376::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70748::DEBUG::2014-09-25 17:37:35,376::task::1191::Storage.TaskManager.Task::(prepare) Task=`507459a6-f186-4426-9ef7-d816329519de`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70748::DEBUG::2014-09-25 17:37:35,377::task::595::Storage.TaskManager.Task::(_updateState) Task=`507459a6-f186-4426-9ef7-d816329519de`::moving from state preparing -> state finished >Thread-70748::DEBUG::2014-09-25 17:37:35,377::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70748::DEBUG::2014-09-25 17:37:35,377::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70748::DEBUG::2014-09-25 17:37:35,377::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70748::DEBUG::2014-09-25 17:37:35,378::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70748::DEBUG::2014-09-25 17:37:35,378::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70748::DEBUG::2014-09-25 17:37:35,378::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70748::DEBUG::2014-09-25 17:37:35,379::task::993::Storage.TaskManager.Task::(_decref) Task=`507459a6-f186-4426-9ef7-d816329519de`::ref 0 aborting False >Thread-70748::DEBUG::2014-09-25 17:37:35,379::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70748::DEBUG::2014-09-25 17:37:35,380::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:36,187::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:36,188::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70749::DEBUG::2014-09-25 17:37:36,190::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:36,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:36,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171586 s, 59.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:37,396::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:37,396::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:37,396::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:37,396::__init__::232::IOProcess::(_processLogs) (108842) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:37,397::__init__::232::IOProcess::(_processLogs) (108842) Queuing response >Thread-79::DEBUG::2014-09-25 17:37:37,398::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:37:37,407::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00034869 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:37,408::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:37,408::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:37,408::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:37,409::__init__::232::IOProcess::(_processLogs) (108843) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:37,409::__init__::232::IOProcess::(_processLogs) (108843) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:37,411::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:37,411::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:37,412::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:37,412::__init__::232::IOProcess::(_processLogs) (108844) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:37,412::__init__::232::IOProcess::(_processLogs) (108844) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:37,413::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:37,413::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:37,414::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:37,414::__init__::232::IOProcess::(_processLogs) (108845) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:37,414::__init__::232::IOProcess::(_processLogs) (108845) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:37,415::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:37,416::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:37,416::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:37,417::__init__::232::IOProcess::(_processLogs) (108846) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:37,417::__init__::232::IOProcess::(_processLogs) (108846) Queuing response >Dummy-148::DEBUG::2014-09-25 17:37:38,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:39,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144357 s, 70.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:39,202::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:39,204::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70750::DEBUG::2014-09-25 17:37:39,206::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:41,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:41,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160043 s, 64.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:42,098::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:42,098::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:42,098::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:42,099::__init__::232::IOProcess::(_processLogs) (108847) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:42,101::__init__::232::IOProcess::(_processLogs) (108847) Queuing response >Thread-81::DEBUG::2014-09-25 17:37:42,102::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:37:42,119::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00899434 s, 41.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:42,120::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:42,121::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:42,121::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:42,121::__init__::232::IOProcess::(_processLogs) (108848) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:42,122::__init__::232::IOProcess::(_processLogs) (108848) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:42,221::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:42,222::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70751::DEBUG::2014-09-25 17:37:42,224::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:43,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:43,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133945 s, 76.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:37:45,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:45,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177136 s, 57.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:45,177::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:45,178::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:45,179::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:45,180::__init__::232::IOProcess::(_processLogs) (108849) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:45,182::__init__::232::IOProcess::(_processLogs) (108849) Queuing response >Thread-80::DEBUG::2014-09-25 17:37:45,184::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:37:45,194::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000409813 s, 922 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:45,197::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:45,198::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:45,199::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:45,200::__init__::232::IOProcess::(_processLogs) (108850) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:45,202::__init__::232::IOProcess::(_processLogs) (108850) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:45,238::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:45,239::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70752::DEBUG::2014-09-25 17:37:45,241::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:45,427::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:45,428::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70753::DEBUG::2014-09-25 17:37:45,429::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70753::DEBUG::2014-09-25 17:37:45,431::task::595::Storage.TaskManager.Task::(_updateState) Task=`244e5643-e098-438c-ab6a-d81391ab784a`::moving from state init -> state preparing >Thread-70753::INFO::2014-09-25 17:37:45,431::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70753::INFO::2014-09-25 17:37:45,431::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70753::DEBUG::2014-09-25 17:37:45,432::task::1191::Storage.TaskManager.Task::(prepare) Task=`244e5643-e098-438c-ab6a-d81391ab784a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70753::DEBUG::2014-09-25 17:37:45,432::task::595::Storage.TaskManager.Task::(_updateState) Task=`244e5643-e098-438c-ab6a-d81391ab784a`::moving from state preparing -> state finished >Thread-70753::DEBUG::2014-09-25 17:37:45,432::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70753::DEBUG::2014-09-25 17:37:45,432::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70753::DEBUG::2014-09-25 17:37:45,433::task::993::Storage.TaskManager.Task::(_decref) Task=`244e5643-e098-438c-ab6a-d81391ab784a`::ref 0 aborting False >Thread-70753::DEBUG::2014-09-25 17:37:45,433::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70753::DEBUG::2014-09-25 17:37:45,433::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:45,446::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:45,447::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70754::DEBUG::2014-09-25 17:37:45,447::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70754::DEBUG::2014-09-25 17:37:45,449::task::595::Storage.TaskManager.Task::(_updateState) Task=`64834ddb-d6f9-4abd-964a-f9674b714ba4`::moving from state init -> state preparing >Thread-70754::INFO::2014-09-25 17:37:45,449::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70754::DEBUG::2014-09-25 17:37:45,450::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`24a88424-d301-431d-8ed3-37f8dbe008a4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70754::DEBUG::2014-09-25 17:37:45,450::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70754::DEBUG::2014-09-25 17:37:45,451::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70754::DEBUG::2014-09-25 17:37:45,451::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`24a88424-d301-431d-8ed3-37f8dbe008a4`::Granted request >Thread-70754::DEBUG::2014-09-25 17:37:45,452::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`64834ddb-d6f9-4abd-964a-f9674b714ba4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70754::DEBUG::2014-09-25 17:37:45,452::task::993::Storage.TaskManager.Task::(_decref) Task=`64834ddb-d6f9-4abd-964a-f9674b714ba4`::ref 1 aborting False >Thread-70754::INFO::2014-09-25 17:37:45,457::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70754::DEBUG::2014-09-25 17:37:45,457::task::1191::Storage.TaskManager.Task::(prepare) Task=`64834ddb-d6f9-4abd-964a-f9674b714ba4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70754::DEBUG::2014-09-25 17:37:45,457::task::595::Storage.TaskManager.Task::(_updateState) Task=`64834ddb-d6f9-4abd-964a-f9674b714ba4`::moving from state preparing -> state finished >Thread-70754::DEBUG::2014-09-25 17:37:45,457::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70754::DEBUG::2014-09-25 17:37:45,458::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70754::DEBUG::2014-09-25 17:37:45,458::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70754::DEBUG::2014-09-25 17:37:45,458::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70754::DEBUG::2014-09-25 17:37:45,459::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70754::DEBUG::2014-09-25 17:37:45,459::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70754::DEBUG::2014-09-25 17:37:45,459::task::993::Storage.TaskManager.Task::(_decref) Task=`64834ddb-d6f9-4abd-964a-f9674b714ba4`::ref 0 aborting False >Thread-70754::DEBUG::2014-09-25 17:37:45,460::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70754::DEBUG::2014-09-25 17:37:45,460::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:47,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:47,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135492 s, 75.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:47,417::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:47,417::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:47,417::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:47,417::__init__::232::IOProcess::(_processLogs) (108851) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:47,426::__init__::232::IOProcess::(_processLogs) (108851) Queuing response >Thread-79::DEBUG::2014-09-25 17:37:47,427::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:37:47,441::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00691255 s, 81.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:47,442::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:47,442::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:47,442::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:47,442::__init__::232::IOProcess::(_processLogs) (108852) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:47,445::__init__::232::IOProcess::(_processLogs) (108852) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:47,446::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:47,446::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:47,446::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:47,447::__init__::232::IOProcess::(_processLogs) (108853) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:47,450::__init__::232::IOProcess::(_processLogs) (108853) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:47,451::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:47,451::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:47,451::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:47,451::__init__::232::IOProcess::(_processLogs) (108854) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:47,457::__init__::232::IOProcess::(_processLogs) (108854) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:47,458::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:47,458::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:47,458::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:47,458::__init__::232::IOProcess::(_processLogs) (108855) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:47,460::__init__::232::IOProcess::(_processLogs) (108855) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:48,254::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:48,255::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70755::DEBUG::2014-09-25 17:37:48,257::task::595::Storage.TaskManager.Task::(_updateState) Task=`231b5269-6a19-4ce0-9fd3-190f5e786db2`::moving from state init -> state preparing >Thread-70755::INFO::2014-09-25 17:37:48,258::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70755::INFO::2014-09-25 17:37:48,258::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00691255', 'lastCheck': '0.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409813', 'lastCheck': '3.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00899434', 'lastCheck': '6.1', 'valid': True}} >Thread-70755::DEBUG::2014-09-25 17:37:48,259::task::1191::Storage.TaskManager.Task::(prepare) Task=`231b5269-6a19-4ce0-9fd3-190f5e786db2`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00691255', 'lastCheck': '0.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409813', 'lastCheck': '3.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00899434', 'lastCheck': '6.1', 'valid': True}} >Thread-70755::DEBUG::2014-09-25 17:37:48,259::task::595::Storage.TaskManager.Task::(_updateState) Task=`231b5269-6a19-4ce0-9fd3-190f5e786db2`::moving from state preparing -> state finished >Thread-70755::DEBUG::2014-09-25 17:37:48,259::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70755::DEBUG::2014-09-25 17:37:48,259::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70755::DEBUG::2014-09-25 17:37:48,260::task::993::Storage.TaskManager.Task::(_decref) Task=`231b5269-6a19-4ce0-9fd3-190f5e786db2`::ref 0 aborting False >Thread-70755::DEBUG::2014-09-25 17:37:48,262::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:48,282::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:48,283::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70756::DEBUG::2014-09-25 17:37:48,285::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:49,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:49,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163263 s, 62.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:51,345::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:51,347::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70757::DEBUG::2014-09-25 17:37:51,349::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:51,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:51,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141182 s, 72.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:52,124::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:52,124::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:52,125::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:52,125::__init__::232::IOProcess::(_processLogs) (108856) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:52,128::__init__::232::IOProcess::(_processLogs) (108856) Queuing response >Thread-81::DEBUG::2014-09-25 17:37:52,129::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:37:52,146::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00808805 s, 45.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:52,147::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:52,148::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:52,148::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:52,148::__init__::232::IOProcess::(_processLogs) (108857) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:52,149::__init__::232::IOProcess::(_processLogs) (108857) Queuing response >Dummy-148::DEBUG::2014-09-25 17:37:53,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:53,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136151 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:54,366::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:54,368::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70758::DEBUG::2014-09-25 17:37:54,370::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:37:55,205::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:55,205::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:55,205::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:55,206::__init__::232::IOProcess::(_processLogs) (108858) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:55,206::__init__::232::IOProcess::(_processLogs) (108858) Queuing response >Thread-80::DEBUG::2014-09-25 17:37:55,207::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:37:55,216::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00052088 s, 726 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:55,217::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:55,218::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:55,218::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:55,218::__init__::232::IOProcess::(_processLogs) (108859) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:55,219::__init__::232::IOProcess::(_processLogs) (108859) Queuing response >Dummy-148::DEBUG::2014-09-25 17:37:55,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:55,497::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:55,499::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70759::DEBUG::2014-09-25 17:37:55,499::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70759::DEBUG::2014-09-25 17:37:55,501::task::595::Storage.TaskManager.Task::(_updateState) Task=`e43757ad-7fe0-444b-984f-79e6ff1fc79e`::moving from state init -> state preparing >Thread-70759::INFO::2014-09-25 17:37:55,501::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70759::INFO::2014-09-25 17:37:55,502::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70759::DEBUG::2014-09-25 17:37:55,502::task::1191::Storage.TaskManager.Task::(prepare) Task=`e43757ad-7fe0-444b-984f-79e6ff1fc79e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70759::DEBUG::2014-09-25 17:37:55,502::task::595::Storage.TaskManager.Task::(_updateState) Task=`e43757ad-7fe0-444b-984f-79e6ff1fc79e`::moving from state preparing -> state finished >Thread-70759::DEBUG::2014-09-25 17:37:55,503::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70759::DEBUG::2014-09-25 17:37:55,503::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70759::DEBUG::2014-09-25 17:37:55,503::task::993::Storage.TaskManager.Task::(_decref) Task=`e43757ad-7fe0-444b-984f-79e6ff1fc79e`::ref 0 aborting False >Thread-70759::DEBUG::2014-09-25 17:37:55,503::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70759::DEBUG::2014-09-25 17:37:55,504::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:37:55,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0236333 s, 43.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:55,518::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:55,521::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70760::DEBUG::2014-09-25 17:37:55,521::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70760::DEBUG::2014-09-25 17:37:55,525::task::595::Storage.TaskManager.Task::(_updateState) Task=`28a55c85-4f22-4ce3-ba68-963c94f9eb90`::moving from state init -> state preparing >Thread-70760::INFO::2014-09-25 17:37:55,525::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70760::DEBUG::2014-09-25 17:37:55,528::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`15271f4a-1f57-4fc9-a91c-d2ef4e9f68a1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70760::DEBUG::2014-09-25 17:37:55,528::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70760::DEBUG::2014-09-25 17:37:55,530::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70760::DEBUG::2014-09-25 17:37:55,530::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`15271f4a-1f57-4fc9-a91c-d2ef4e9f68a1`::Granted request >Thread-70760::DEBUG::2014-09-25 17:37:55,532::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`28a55c85-4f22-4ce3-ba68-963c94f9eb90`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70760::DEBUG::2014-09-25 17:37:55,533::task::993::Storage.TaskManager.Task::(_decref) Task=`28a55c85-4f22-4ce3-ba68-963c94f9eb90`::ref 1 aborting False >Thread-70760::INFO::2014-09-25 17:37:55,548::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70760::DEBUG::2014-09-25 17:37:55,549::task::1191::Storage.TaskManager.Task::(prepare) Task=`28a55c85-4f22-4ce3-ba68-963c94f9eb90`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70760::DEBUG::2014-09-25 17:37:55,550::task::595::Storage.TaskManager.Task::(_updateState) Task=`28a55c85-4f22-4ce3-ba68-963c94f9eb90`::moving from state preparing -> state finished >Thread-70760::DEBUG::2014-09-25 17:37:55,551::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70760::DEBUG::2014-09-25 17:37:55,551::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70760::DEBUG::2014-09-25 17:37:55,553::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70760::DEBUG::2014-09-25 17:37:55,554::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70760::DEBUG::2014-09-25 17:37:55,555::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70760::DEBUG::2014-09-25 17:37:55,555::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70760::DEBUG::2014-09-25 17:37:55,557::task::993::Storage.TaskManager.Task::(_decref) Task=`28a55c85-4f22-4ce3-ba68-963c94f9eb90`::ref 0 aborting False >Thread-70760::DEBUG::2014-09-25 17:37:55,559::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70760::DEBUG::2014-09-25 17:37:55,560::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:37:57,388::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:37:57,390::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70761::DEBUG::2014-09-25 17:37:57,392::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:37:57,462::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:57,462::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:57,463::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:57,463::__init__::232::IOProcess::(_processLogs) (108860) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:57,468::__init__::232::IOProcess::(_processLogs) (108860) Queuing response >Thread-79::DEBUG::2014-09-25 17:37:57,469::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:37:57,490::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00290074 s, 194 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:37:57,492::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:57,492::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:57,492::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:57,493::__init__::232::IOProcess::(_processLogs) (108861) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:37:57,495::__init__::232::IOProcess::(_processLogs) (108861) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:57,496::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:57,496::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:57,497::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:57,497::__init__::232::IOProcess::(_processLogs) (108862) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:57,497::__init__::232::IOProcess::(_processLogs) (108862) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:57,498::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:57,499::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:57,499::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:57,499::__init__::232::IOProcess::(_processLogs) (108863) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:57,500::__init__::232::IOProcess::(_processLogs) (108863) Queuing response >Thread-29::DEBUG::2014-09-25 17:37:57,500::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:37:57,500::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:37:57,501::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:37:57,501::__init__::232::IOProcess::(_processLogs) (108864) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:37:57,501::__init__::232::IOProcess::(_processLogs) (108864) Queuing response >Dummy-148::DEBUG::2014-09-25 17:37:57,579::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:57,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018628 s, 55.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:37:59,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:37:59,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134077 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:00,408::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:00,409::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70762::DEBUG::2014-09-25 17:38:00,411::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:01,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:01,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132983 s, 77.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:02,152::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:02,153::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:02,155::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:02,156::__init__::232::IOProcess::(_processLogs) (108865) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:02,164::__init__::232::IOProcess::(_processLogs) (108865) Queuing response >Thread-81::DEBUG::2014-09-25 17:38:02,165::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:38:02,193::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00654677 s, 56.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:02,195::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:02,195::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:02,195::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:02,195::__init__::232::IOProcess::(_processLogs) (108866) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:02,198::__init__::232::IOProcess::(_processLogs) (108866) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:03,424::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:03,426::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70763::DEBUG::2014-09-25 17:38:03,428::task::595::Storage.TaskManager.Task::(_updateState) Task=`713ce20b-255e-47d9-aa5a-3839b61054f9`::moving from state init -> state preparing >Thread-70763::INFO::2014-09-25 17:38:03,428::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70763::INFO::2014-09-25 17:38:03,429::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00290074', 'lastCheck': '5.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00052088', 'lastCheck': '8.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00654677', 'lastCheck': '1.2', 'valid': True}} >Thread-70763::DEBUG::2014-09-25 17:38:03,429::task::1191::Storage.TaskManager.Task::(prepare) Task=`713ce20b-255e-47d9-aa5a-3839b61054f9`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00290074', 'lastCheck': '5.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00052088', 'lastCheck': '8.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00654677', 'lastCheck': '1.2', 'valid': True}} >Thread-70763::DEBUG::2014-09-25 17:38:03,430::task::595::Storage.TaskManager.Task::(_updateState) Task=`713ce20b-255e-47d9-aa5a-3839b61054f9`::moving from state preparing -> state finished >Thread-70763::DEBUG::2014-09-25 17:38:03,430::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70763::DEBUG::2014-09-25 17:38:03,430::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70763::DEBUG::2014-09-25 17:38:03,430::task::993::Storage.TaskManager.Task::(_decref) Task=`713ce20b-255e-47d9-aa5a-3839b61054f9`::ref 0 aborting False >Thread-70763::DEBUG::2014-09-25 17:38:03,433::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:03,453::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:03,454::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70764::DEBUG::2014-09-25 17:38:03,456::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:03,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:03,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.22049 s, 4.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:05,221::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:05,221::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:05,222::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:05,222::__init__::232::IOProcess::(_processLogs) (108867) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:05,222::__init__::232::IOProcess::(_processLogs) (108867) Queuing response >Thread-80::DEBUG::2014-09-25 17:38:05,223::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:38:05,237::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000540051 s, 700 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:05,239::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:05,239::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:05,239::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:05,239::__init__::232::IOProcess::(_processLogs) (108868) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:05,240::__init__::232::IOProcess::(_processLogs) (108868) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:05,592::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70765::DEBUG::2014-09-25 17:38:05,593::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:38:05,593::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70765::DEBUG::2014-09-25 17:38:05,595::task::595::Storage.TaskManager.Task::(_updateState) Task=`e0c7f3fa-6e10-4a67-93c3-84037e757c68`::moving from state init -> state preparing >Thread-70765::INFO::2014-09-25 17:38:05,596::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70765::INFO::2014-09-25 17:38:05,596::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70765::DEBUG::2014-09-25 17:38:05,596::task::1191::Storage.TaskManager.Task::(prepare) Task=`e0c7f3fa-6e10-4a67-93c3-84037e757c68`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70765::DEBUG::2014-09-25 17:38:05,597::task::595::Storage.TaskManager.Task::(_updateState) Task=`e0c7f3fa-6e10-4a67-93c3-84037e757c68`::moving from state preparing -> state finished >Thread-70765::DEBUG::2014-09-25 17:38:05,597::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70765::DEBUG::2014-09-25 17:38:05,597::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70765::DEBUG::2014-09-25 17:38:05,597::task::993::Storage.TaskManager.Task::(_decref) Task=`e0c7f3fa-6e10-4a67-93c3-84037e757c68`::ref 0 aborting False >Thread-70765::DEBUG::2014-09-25 17:38:05,598::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70765::DEBUG::2014-09-25 17:38:05,598::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:05,614::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:05,615::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70766::DEBUG::2014-09-25 17:38:05,615::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70766::DEBUG::2014-09-25 17:38:05,617::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d3083fe-01ac-4a26-9407-f6b630a8222e`::moving from state init -> state preparing >Thread-70766::INFO::2014-09-25 17:38:05,617::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70766::DEBUG::2014-09-25 17:38:05,618::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3aef7e58-394a-4dfc-afdb-63ab7b141515`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70766::DEBUG::2014-09-25 17:38:05,618::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70766::DEBUG::2014-09-25 17:38:05,618::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70766::DEBUG::2014-09-25 17:38:05,619::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3aef7e58-394a-4dfc-afdb-63ab7b141515`::Granted request >Thread-70766::DEBUG::2014-09-25 17:38:05,619::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7d3083fe-01ac-4a26-9407-f6b630a8222e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70766::DEBUG::2014-09-25 17:38:05,620::task::993::Storage.TaskManager.Task::(_decref) Task=`7d3083fe-01ac-4a26-9407-f6b630a8222e`::ref 1 aborting False >Thread-70766::INFO::2014-09-25 17:38:05,624::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70766::DEBUG::2014-09-25 17:38:05,624::task::1191::Storage.TaskManager.Task::(prepare) Task=`7d3083fe-01ac-4a26-9407-f6b630a8222e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70766::DEBUG::2014-09-25 17:38:05,624::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d3083fe-01ac-4a26-9407-f6b630a8222e`::moving from state preparing -> state finished >Thread-70766::DEBUG::2014-09-25 17:38:05,625::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70766::DEBUG::2014-09-25 17:38:05,625::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70766::DEBUG::2014-09-25 17:38:05,625::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70766::DEBUG::2014-09-25 17:38:05,626::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70766::DEBUG::2014-09-25 17:38:05,626::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70766::DEBUG::2014-09-25 17:38:05,626::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70766::DEBUG::2014-09-25 17:38:05,626::task::993::Storage.TaskManager.Task::(_decref) Task=`7d3083fe-01ac-4a26-9407-f6b630a8222e`::ref 0 aborting False >Thread-70766::DEBUG::2014-09-25 17:38:05,627::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70766::DEBUG::2014-09-25 17:38:05,627::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:05,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:05,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180831 s, 56.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:06,521::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:06,522::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70767::DEBUG::2014-09-25 17:38:06,524::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:38:07,503::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:07,504::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:07,504::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:07,504::__init__::232::IOProcess::(_processLogs) (108869) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:07,506::__init__::232::IOProcess::(_processLogs) (108869) Queuing response >Thread-79::DEBUG::2014-09-25 17:38:07,507::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:38:07,526::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0027956 s, 201 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:07,527::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:07,528::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:07,528::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:07,528::__init__::232::IOProcess::(_processLogs) (108870) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:07,529::__init__::232::IOProcess::(_processLogs) (108870) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:07,530::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:07,531::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:07,531::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:07,531::__init__::232::IOProcess::(_processLogs) (108871) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:07,532::__init__::232::IOProcess::(_processLogs) (108871) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:07,532::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:07,532::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:07,533::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:07,534::__init__::232::IOProcess::(_processLogs) (108872) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:07,534::__init__::232::IOProcess::(_processLogs) (108872) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:07,534::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:07,535::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:07,535::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:07,535::__init__::232::IOProcess::(_processLogs) (108873) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:07,535::__init__::232::IOProcess::(_processLogs) (108873) Queuing response >Dummy-148::DEBUG::2014-09-25 17:38:08,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:08,047::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214492 s, 47.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:09,543::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:09,545::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70768::DEBUG::2014-09-25 17:38:09,547::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:10,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:10,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166052 s, 61.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:38:12,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:12,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170116 s, 60.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:12,201::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:12,201::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:12,201::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:12,201::__init__::232::IOProcess::(_processLogs) (108874) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:12,203::__init__::232::IOProcess::(_processLogs) (108874) Queuing response >Thread-81::DEBUG::2014-09-25 17:38:12,204::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:38:12,218::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00305322 s, 122 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:12,219::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:12,219::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:12,220::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:12,220::__init__::232::IOProcess::(_processLogs) (108875) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:12,223::__init__::232::IOProcess::(_processLogs) (108875) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:12,563::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:12,564::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70769::DEBUG::2014-09-25 17:38:12,566::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:14,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:14,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0299054 s, 34.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:15,242::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:15,243::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:15,243::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:38:15,244::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:38:15,245::__init__::232::IOProcess::(_processLogs) (108876) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:15,254::__init__::232::IOProcess::(_processLogs) (108876) Queuing response >Thread-80::DEBUG::2014-09-25 17:38:15,255::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000475384 s, 795 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:15,256::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:15,256::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:15,256::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:15,256::__init__::232::IOProcess::(_processLogs) (108877) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:15,257::__init__::232::IOProcess::(_processLogs) (108877) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:15,582::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:15,583::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70770::DEBUG::2014-09-25 17:38:15,585::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:15,655::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70771::DEBUG::2014-09-25 17:38:15,656::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:38:15,656::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70771::DEBUG::2014-09-25 17:38:15,659::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b63fb37-8acb-48ca-99cd-95a42869d23a`::moving from state init -> state preparing >Thread-70771::INFO::2014-09-25 17:38:15,659::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70771::INFO::2014-09-25 17:38:15,660::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70771::DEBUG::2014-09-25 17:38:15,660::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b63fb37-8acb-48ca-99cd-95a42869d23a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70771::DEBUG::2014-09-25 17:38:15,660::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b63fb37-8acb-48ca-99cd-95a42869d23a`::moving from state preparing -> state finished >Thread-70771::DEBUG::2014-09-25 17:38:15,660::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70771::DEBUG::2014-09-25 17:38:15,661::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70771::DEBUG::2014-09-25 17:38:15,661::task::993::Storage.TaskManager.Task::(_decref) Task=`1b63fb37-8acb-48ca-99cd-95a42869d23a`::ref 0 aborting False >Thread-70771::DEBUG::2014-09-25 17:38:15,661::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70771::DEBUG::2014-09-25 17:38:15,662::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:15,675::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70772::DEBUG::2014-09-25 17:38:15,676::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:38:15,677::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70772::DEBUG::2014-09-25 17:38:15,679::task::595::Storage.TaskManager.Task::(_updateState) Task=`cae4dc87-ed98-4191-b92c-5863031a401e`::moving from state init -> state preparing >Thread-70772::INFO::2014-09-25 17:38:15,679::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70772::DEBUG::2014-09-25 17:38:15,680::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`eac3fb36-2f64-4445-8cc4-21cf4d2858ea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70772::DEBUG::2014-09-25 17:38:15,680::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70772::DEBUG::2014-09-25 17:38:15,681::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70772::DEBUG::2014-09-25 17:38:15,681::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`eac3fb36-2f64-4445-8cc4-21cf4d2858ea`::Granted request >Thread-70772::DEBUG::2014-09-25 17:38:15,681::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`cae4dc87-ed98-4191-b92c-5863031a401e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70772::DEBUG::2014-09-25 17:38:15,682::task::993::Storage.TaskManager.Task::(_decref) Task=`cae4dc87-ed98-4191-b92c-5863031a401e`::ref 1 aborting False >Thread-70772::INFO::2014-09-25 17:38:15,686::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70772::DEBUG::2014-09-25 17:38:15,686::task::1191::Storage.TaskManager.Task::(prepare) Task=`cae4dc87-ed98-4191-b92c-5863031a401e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70772::DEBUG::2014-09-25 17:38:15,687::task::595::Storage.TaskManager.Task::(_updateState) Task=`cae4dc87-ed98-4191-b92c-5863031a401e`::moving from state preparing -> state finished >Thread-70772::DEBUG::2014-09-25 17:38:15,687::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70772::DEBUG::2014-09-25 17:38:15,687::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70772::DEBUG::2014-09-25 17:38:15,688::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70772::DEBUG::2014-09-25 17:38:15,688::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70772::DEBUG::2014-09-25 17:38:15,688::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70772::DEBUG::2014-09-25 17:38:15,688::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70772::DEBUG::2014-09-25 17:38:15,689::task::993::Storage.TaskManager.Task::(_decref) Task=`cae4dc87-ed98-4191-b92c-5863031a401e`::ref 0 aborting False >Thread-70772::DEBUG::2014-09-25 17:38:15,689::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70772::DEBUG::2014-09-25 17:38:15,690::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:16,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:16,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0306724 s, 33.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:17,537::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:17,537::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:17,538::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:17,538::__init__::232::IOProcess::(_processLogs) (108878) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:17,542::__init__::232::IOProcess::(_processLogs) (108878) Queuing response >Thread-79::DEBUG::2014-09-25 17:38:17,543::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:38:17,558::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00556537 s, 101 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:17,559::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:17,559::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:17,559::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:17,560::__init__::232::IOProcess::(_processLogs) (108879) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:17,560::__init__::232::IOProcess::(_processLogs) (108879) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:17,561::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:17,562::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:17,563::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:17,563::__init__::232::IOProcess::(_processLogs) (108880) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:17,563::__init__::232::IOProcess::(_processLogs) (108880) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:17,564::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:17,564::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:17,564::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:17,564::__init__::232::IOProcess::(_processLogs) (108881) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:17,564::__init__::232::IOProcess::(_processLogs) (108881) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:17,565::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:17,566::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:17,566::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:17,566::__init__::232::IOProcess::(_processLogs) (108882) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:17,566::__init__::232::IOProcess::(_processLogs) (108882) Queuing response >Dummy-148::DEBUG::2014-09-25 17:38:18,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:18,527::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164849 s, 62.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:18,600::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:18,602::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70773::DEBUG::2014-09-25 17:38:18,605::task::595::Storage.TaskManager.Task::(_updateState) Task=`eca4c7f5-cae8-4a25-881c-0188081be696`::moving from state init -> state preparing >Thread-70773::INFO::2014-09-25 17:38:18,605::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70773::INFO::2014-09-25 17:38:18,606::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00556537', 'lastCheck': '1.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000475384', 'lastCheck': '3.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00305322', 'lastCheck': '6.4', 'valid': True}} >Thread-70773::DEBUG::2014-09-25 17:38:18,606::task::1191::Storage.TaskManager.Task::(prepare) Task=`eca4c7f5-cae8-4a25-881c-0188081be696`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00556537', 'lastCheck': '1.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000475384', 'lastCheck': '3.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00305322', 'lastCheck': '6.4', 'valid': True}} >Thread-70773::DEBUG::2014-09-25 17:38:18,606::task::595::Storage.TaskManager.Task::(_updateState) Task=`eca4c7f5-cae8-4a25-881c-0188081be696`::moving from state preparing -> state finished >Thread-70773::DEBUG::2014-09-25 17:38:18,606::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70773::DEBUG::2014-09-25 17:38:18,607::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70773::DEBUG::2014-09-25 17:38:18,607::task::993::Storage.TaskManager.Task::(_decref) Task=`eca4c7f5-cae8-4a25-881c-0188081be696`::ref 0 aborting False >Thread-70773::DEBUG::2014-09-25 17:38:18,609::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:18,632::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:18,633::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70774::DEBUG::2014-09-25 17:38:18,635::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:20,546::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:20,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156217 s, 65.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:21,695::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:21,697::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70775::DEBUG::2014-09-25 17:38:21,700::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:38:22,225::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:22,225::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:22,225::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:22,226::__init__::232::IOProcess::(_processLogs) (108883) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:22,230::__init__::232::IOProcess::(_processLogs) (108883) Queuing response >Thread-81::DEBUG::2014-09-25 17:38:22,232::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:38:22,247::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00211272 s, 176 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:22,248::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:22,249::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:22,250::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:22,251::__init__::232::IOProcess::(_processLogs) (108884) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:22,254::__init__::232::IOProcess::(_processLogs) (108884) Queuing response >Dummy-148::DEBUG::2014-09-25 17:38:22,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:22,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192473 s, 53.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:38:24,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:24,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172359 s, 59.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:24,721::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:24,723::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70776::DEBUG::2014-09-25 17:38:24,725::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:38:25,259::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:25,259::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:25,260::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:25,260::__init__::232::IOProcess::(_processLogs) (108885) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:25,260::__init__::232::IOProcess::(_processLogs) (108885) Queuing response >Thread-80::DEBUG::2014-09-25 17:38:25,262::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:38:25,270::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000416041 s, 909 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:25,272::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:25,272::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:25,272::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:25,273::__init__::232::IOProcess::(_processLogs) (108886) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:25,273::__init__::232::IOProcess::(_processLogs) (108886) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:25,718::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70777::DEBUG::2014-09-25 17:38:25,719::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:38:25,719::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70777::DEBUG::2014-09-25 17:38:25,721::task::595::Storage.TaskManager.Task::(_updateState) Task=`247127bc-b3fe-48ff-9491-9292925734b2`::moving from state init -> state preparing >Thread-70777::INFO::2014-09-25 17:38:25,722::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70777::INFO::2014-09-25 17:38:25,722::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70777::DEBUG::2014-09-25 17:38:25,723::task::1191::Storage.TaskManager.Task::(prepare) Task=`247127bc-b3fe-48ff-9491-9292925734b2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70777::DEBUG::2014-09-25 17:38:25,723::task::595::Storage.TaskManager.Task::(_updateState) Task=`247127bc-b3fe-48ff-9491-9292925734b2`::moving from state preparing -> state finished >Thread-70777::DEBUG::2014-09-25 17:38:25,723::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70777::DEBUG::2014-09-25 17:38:25,723::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70777::DEBUG::2014-09-25 17:38:25,723::task::993::Storage.TaskManager.Task::(_decref) Task=`247127bc-b3fe-48ff-9491-9292925734b2`::ref 0 aborting False >Thread-70777::DEBUG::2014-09-25 17:38:25,724::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70777::DEBUG::2014-09-25 17:38:25,724::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:25,737::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:25,738::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70778::DEBUG::2014-09-25 17:38:25,738::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70778::DEBUG::2014-09-25 17:38:25,740::task::595::Storage.TaskManager.Task::(_updateState) Task=`3167821f-5f61-4739-b26f-856c378cc9a3`::moving from state init -> state preparing >Thread-70778::INFO::2014-09-25 17:38:25,740::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70778::DEBUG::2014-09-25 17:38:25,741::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2018f107-fe92-4907-b439-e38454e114e6`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70778::DEBUG::2014-09-25 17:38:25,741::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70778::DEBUG::2014-09-25 17:38:25,742::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70778::DEBUG::2014-09-25 17:38:25,742::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2018f107-fe92-4907-b439-e38454e114e6`::Granted request >Thread-70778::DEBUG::2014-09-25 17:38:25,743::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3167821f-5f61-4739-b26f-856c378cc9a3`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70778::DEBUG::2014-09-25 17:38:25,743::task::993::Storage.TaskManager.Task::(_decref) Task=`3167821f-5f61-4739-b26f-856c378cc9a3`::ref 1 aborting False >Thread-70778::INFO::2014-09-25 17:38:25,747::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70778::DEBUG::2014-09-25 17:38:25,747::task::1191::Storage.TaskManager.Task::(prepare) Task=`3167821f-5f61-4739-b26f-856c378cc9a3`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70778::DEBUG::2014-09-25 17:38:25,748::task::595::Storage.TaskManager.Task::(_updateState) Task=`3167821f-5f61-4739-b26f-856c378cc9a3`::moving from state preparing -> state finished >Thread-70778::DEBUG::2014-09-25 17:38:25,748::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70778::DEBUG::2014-09-25 17:38:25,748::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70778::DEBUG::2014-09-25 17:38:25,749::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70778::DEBUG::2014-09-25 17:38:25,749::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70778::DEBUG::2014-09-25 17:38:25,749::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70778::DEBUG::2014-09-25 17:38:25,749::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70778::DEBUG::2014-09-25 17:38:25,750::task::993::Storage.TaskManager.Task::(_decref) Task=`3167821f-5f61-4739-b26f-856c378cc9a3`::ref 0 aborting False >Thread-70778::DEBUG::2014-09-25 17:38:25,750::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70778::DEBUG::2014-09-25 17:38:25,751::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:26,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:26,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146191 s, 70.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:27,568::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:27,568::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:27,569::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:27,569::__init__::232::IOProcess::(_processLogs) (108887) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:27,573::__init__::232::IOProcess::(_processLogs) (108887) Queuing response >Thread-79::DEBUG::2014-09-25 17:38:27,574::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:38:27,589::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00468567 s, 120 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:27,591::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:27,591::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:27,591::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:27,592::__init__::232::IOProcess::(_processLogs) (108888) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:27,596::__init__::232::IOProcess::(_processLogs) (108888) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:27,597::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:27,597::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:27,598::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:27,599::__init__::232::IOProcess::(_processLogs) (108889) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:27,599::__init__::232::IOProcess::(_processLogs) (108889) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:27,599::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:27,600::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:27,600::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:27,601::__init__::232::IOProcess::(_processLogs) (108890) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:27,601::__init__::232::IOProcess::(_processLogs) (108890) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:27,601::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:27,602::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:27,603::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:27,603::__init__::232::IOProcess::(_processLogs) (108891) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:27,603::__init__::232::IOProcess::(_processLogs) (108891) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:27,742::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:27,744::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70779::DEBUG::2014-09-25 17:38:27,746::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:28,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:28,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207918 s, 49.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:30,762::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:30,763::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70780::DEBUG::2014-09-25 17:38:30,765::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:30,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:30,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0290738 s, 35.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:32,257::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:32,257::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:32,258::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:32,258::__init__::232::IOProcess::(_processLogs) (108892) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:32,262::__init__::232::IOProcess::(_processLogs) (108892) Queuing response >Thread-81::DEBUG::2014-09-25 17:38:32,263::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:38:32,277::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00318215 s, 117 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:32,279::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:32,279::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:32,279::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:32,279::__init__::232::IOProcess::(_processLogs) (108893) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:32,286::__init__::232::IOProcess::(_processLogs) (108893) Queuing response >Dummy-148::DEBUG::2014-09-25 17:38:32,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:32,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132967 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:33,779::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:33,781::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70781::DEBUG::2014-09-25 17:38:33,783::task::595::Storage.TaskManager.Task::(_updateState) Task=`239f2aa6-cfed-4434-a15a-a470dd4ea6c1`::moving from state init -> state preparing >Thread-70781::INFO::2014-09-25 17:38:33,784::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70781::INFO::2014-09-25 17:38:33,784::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00468567', 'lastCheck': '6.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000416041', 'lastCheck': '8.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00318215', 'lastCheck': '1.5', 'valid': True}} >Thread-70781::DEBUG::2014-09-25 17:38:33,784::task::1191::Storage.TaskManager.Task::(prepare) Task=`239f2aa6-cfed-4434-a15a-a470dd4ea6c1`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00468567', 'lastCheck': '6.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000416041', 'lastCheck': '8.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00318215', 'lastCheck': '1.5', 'valid': True}} >Thread-70781::DEBUG::2014-09-25 17:38:33,785::task::595::Storage.TaskManager.Task::(_updateState) Task=`239f2aa6-cfed-4434-a15a-a470dd4ea6c1`::moving from state preparing -> state finished >Thread-70781::DEBUG::2014-09-25 17:38:33,785::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70781::DEBUG::2014-09-25 17:38:33,785::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70781::DEBUG::2014-09-25 17:38:33,785::task::993::Storage.TaskManager.Task::(_decref) Task=`239f2aa6-cfed-4434-a15a-a470dd4ea6c1`::ref 0 aborting False >Thread-70781::DEBUG::2014-09-25 17:38:33,788::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:33,811::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:33,812::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70782::DEBUG::2014-09-25 17:38:33,814::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:34,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:34,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014105 s, 72.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:35,275::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:35,276::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:35,276::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:35,276::__init__::232::IOProcess::(_processLogs) (108894) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:35,277::__init__::232::IOProcess::(_processLogs) (108894) Queuing response >Thread-80::DEBUG::2014-09-25 17:38:35,278::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:38:35,287::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000509324 s, 742 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:35,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:35,288::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:35,289::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:35,289::__init__::232::IOProcess::(_processLogs) (108895) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:35,289::__init__::232::IOProcess::(_processLogs) (108895) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:35,779::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:35,780::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70783::DEBUG::2014-09-25 17:38:35,780::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70783::DEBUG::2014-09-25 17:38:35,782::task::595::Storage.TaskManager.Task::(_updateState) Task=`44907a33-c0d9-459b-84ab-d93fda33ce91`::moving from state init -> state preparing >Thread-70783::INFO::2014-09-25 17:38:35,782::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70783::INFO::2014-09-25 17:38:35,783::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70783::DEBUG::2014-09-25 17:38:35,783::task::1191::Storage.TaskManager.Task::(prepare) Task=`44907a33-c0d9-459b-84ab-d93fda33ce91`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70783::DEBUG::2014-09-25 17:38:35,783::task::595::Storage.TaskManager.Task::(_updateState) Task=`44907a33-c0d9-459b-84ab-d93fda33ce91`::moving from state preparing -> state finished >Thread-70783::DEBUG::2014-09-25 17:38:35,783::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70783::DEBUG::2014-09-25 17:38:35,783::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70783::DEBUG::2014-09-25 17:38:35,784::task::993::Storage.TaskManager.Task::(_decref) Task=`44907a33-c0d9-459b-84ab-d93fda33ce91`::ref 0 aborting False >Thread-70783::DEBUG::2014-09-25 17:38:35,784::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70783::DEBUG::2014-09-25 17:38:35,784::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:35,797::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:35,798::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70784::DEBUG::2014-09-25 17:38:35,798::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70784::DEBUG::2014-09-25 17:38:35,800::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3957e6d-6312-44e5-ba8f-c990932ed4e7`::moving from state init -> state preparing >Thread-70784::INFO::2014-09-25 17:38:35,800::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70784::DEBUG::2014-09-25 17:38:35,801::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3b98f9fa-293d-41b7-8e5d-228d94774bcb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70784::DEBUG::2014-09-25 17:38:35,801::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70784::DEBUG::2014-09-25 17:38:35,802::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70784::DEBUG::2014-09-25 17:38:35,802::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3b98f9fa-293d-41b7-8e5d-228d94774bcb`::Granted request >Thread-70784::DEBUG::2014-09-25 17:38:35,803::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a3957e6d-6312-44e5-ba8f-c990932ed4e7`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70784::DEBUG::2014-09-25 17:38:35,803::task::993::Storage.TaskManager.Task::(_decref) Task=`a3957e6d-6312-44e5-ba8f-c990932ed4e7`::ref 1 aborting False >Thread-70784::INFO::2014-09-25 17:38:35,807::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70784::DEBUG::2014-09-25 17:38:35,807::task::1191::Storage.TaskManager.Task::(prepare) Task=`a3957e6d-6312-44e5-ba8f-c990932ed4e7`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70784::DEBUG::2014-09-25 17:38:35,808::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3957e6d-6312-44e5-ba8f-c990932ed4e7`::moving from state preparing -> state finished >Thread-70784::DEBUG::2014-09-25 17:38:35,808::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70784::DEBUG::2014-09-25 17:38:35,808::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70784::DEBUG::2014-09-25 17:38:35,808::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70784::DEBUG::2014-09-25 17:38:35,809::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70784::DEBUG::2014-09-25 17:38:35,809::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70784::DEBUG::2014-09-25 17:38:35,809::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70784::DEBUG::2014-09-25 17:38:35,810::task::993::Storage.TaskManager.Task::(_decref) Task=`a3957e6d-6312-44e5-ba8f-c990932ed4e7`::ref 0 aborting False >Thread-70784::DEBUG::2014-09-25 17:38:35,810::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70784::DEBUG::2014-09-25 17:38:35,811::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:36,877::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:36,878::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70785::DEBUG::2014-09-25 17:38:36,880::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:36,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:36,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129967 s, 78.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:37,604::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:37,604::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:37,604::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:37,605::__init__::232::IOProcess::(_processLogs) (108896) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:37,611::__init__::232::IOProcess::(_processLogs) (108896) Queuing response >Thread-79::DEBUG::2014-09-25 17:38:37,612::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:38:37,629::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0067034 s, 83.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:37,630::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:37,630::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:37,631::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:37,631::__init__::232::IOProcess::(_processLogs) (108897) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:37,639::__init__::232::IOProcess::(_processLogs) (108897) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:37,640::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:37,640::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:37,641::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:37,642::__init__::232::IOProcess::(_processLogs) (108898) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:37,642::__init__::232::IOProcess::(_processLogs) (108898) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:37,642::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:37,643::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:37,643::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:37,644::__init__::232::IOProcess::(_processLogs) (108899) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:37,644::__init__::232::IOProcess::(_processLogs) (108899) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:37,644::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:37,645::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:37,645::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:37,645::__init__::232::IOProcess::(_processLogs) (108900) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:37,645::__init__::232::IOProcess::(_processLogs) (108900) Queuing response >Dummy-148::DEBUG::2014-09-25 17:38:38,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:39,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201747 s, 50.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:39,895::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:39,897::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70786::DEBUG::2014-09-25 17:38:39,899::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:41,041::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:41,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133318 s, 76.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:42,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:42,288::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:42,288::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:42,289::__init__::232::IOProcess::(_processLogs) (108901) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:42,291::__init__::232::IOProcess::(_processLogs) (108901) Queuing response >Thread-81::DEBUG::2014-09-25 17:38:42,292::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:38:42,307::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00417369 s, 88.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:42,308::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:42,308::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:42,309::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:42,309::__init__::232::IOProcess::(_processLogs) (108902) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:42,313::__init__::232::IOProcess::(_processLogs) (108902) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:42,914::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:42,916::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70787::DEBUG::2014-09-25 17:38:42,918::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:43,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:43,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020965 s, 48.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:38:45,138::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:45,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157743 s, 64.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:45,292::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:45,292::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:45,292::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:45,293::__init__::232::IOProcess::(_processLogs) (108903) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:45,293::__init__::232::IOProcess::(_processLogs) (108903) Queuing response >Thread-80::DEBUG::2014-09-25 17:38:45,295::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:38:45,304::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000392629 s, 963 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:45,305::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:45,306::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:45,306::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:45,306::__init__::232::IOProcess::(_processLogs) (108904) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:45,307::__init__::232::IOProcess::(_processLogs) (108904) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:45,838::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:45,839::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70788::DEBUG::2014-09-25 17:38:45,839::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70788::DEBUG::2014-09-25 17:38:45,841::task::595::Storage.TaskManager.Task::(_updateState) Task=`f1d860e8-3443-4e29-ab4c-07190014bd9e`::moving from state init -> state preparing >Thread-70788::INFO::2014-09-25 17:38:45,841::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70788::INFO::2014-09-25 17:38:45,842::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70788::DEBUG::2014-09-25 17:38:45,842::task::1191::Storage.TaskManager.Task::(prepare) Task=`f1d860e8-3443-4e29-ab4c-07190014bd9e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70788::DEBUG::2014-09-25 17:38:45,842::task::595::Storage.TaskManager.Task::(_updateState) Task=`f1d860e8-3443-4e29-ab4c-07190014bd9e`::moving from state preparing -> state finished >Thread-70788::DEBUG::2014-09-25 17:38:45,842::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70788::DEBUG::2014-09-25 17:38:45,843::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70788::DEBUG::2014-09-25 17:38:45,843::task::993::Storage.TaskManager.Task::(_decref) Task=`f1d860e8-3443-4e29-ab4c-07190014bd9e`::ref 0 aborting False >Thread-70788::DEBUG::2014-09-25 17:38:45,843::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70788::DEBUG::2014-09-25 17:38:45,844::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:45,856::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70789::DEBUG::2014-09-25 17:38:45,858::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:38:45,858::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70789::DEBUG::2014-09-25 17:38:45,860::task::595::Storage.TaskManager.Task::(_updateState) Task=`5b5d3f6c-552a-4109-8a8a-d564e64d9c97`::moving from state init -> state preparing >Thread-70789::INFO::2014-09-25 17:38:45,860::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70789::DEBUG::2014-09-25 17:38:45,861::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0bcf4171-6ab7-4fa7-b395-1db18093ec94`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70789::DEBUG::2014-09-25 17:38:45,861::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70789::DEBUG::2014-09-25 17:38:45,862::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70789::DEBUG::2014-09-25 17:38:45,862::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0bcf4171-6ab7-4fa7-b395-1db18093ec94`::Granted request >Thread-70789::DEBUG::2014-09-25 17:38:45,863::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5b5d3f6c-552a-4109-8a8a-d564e64d9c97`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70789::DEBUG::2014-09-25 17:38:45,863::task::993::Storage.TaskManager.Task::(_decref) Task=`5b5d3f6c-552a-4109-8a8a-d564e64d9c97`::ref 1 aborting False >Thread-70789::INFO::2014-09-25 17:38:45,868::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70789::DEBUG::2014-09-25 17:38:45,868::task::1191::Storage.TaskManager.Task::(prepare) Task=`5b5d3f6c-552a-4109-8a8a-d564e64d9c97`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70789::DEBUG::2014-09-25 17:38:45,868::task::595::Storage.TaskManager.Task::(_updateState) Task=`5b5d3f6c-552a-4109-8a8a-d564e64d9c97`::moving from state preparing -> state finished >Thread-70789::DEBUG::2014-09-25 17:38:45,868::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70789::DEBUG::2014-09-25 17:38:45,869::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70789::DEBUG::2014-09-25 17:38:45,869::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70789::DEBUG::2014-09-25 17:38:45,869::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70789::DEBUG::2014-09-25 17:38:45,870::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70789::DEBUG::2014-09-25 17:38:45,870::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70789::DEBUG::2014-09-25 17:38:45,870::task::993::Storage.TaskManager.Task::(_decref) Task=`5b5d3f6c-552a-4109-8a8a-d564e64d9c97`::ref 0 aborting False >Thread-70789::DEBUG::2014-09-25 17:38:45,871::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70789::DEBUG::2014-09-25 17:38:45,871::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:45,932::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:45,933::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70790::DEBUG::2014-09-25 17:38:45,935::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:47,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:47,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213416 s, 48.0 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 17:38:47,646::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:38:47,650::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,651::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,651::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,651::__init__::232::IOProcess::(_processLogs) (108905) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:38:47,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,652::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,652::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,653::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,653::__init__::232::IOProcess::(_processLogs) (108906) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:38:47,653::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,653::__init__::232::IOProcess::(_processLogs) (108907) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:38:47,654::__init__::232::IOProcess::(_processLogs) (108906) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:47,655::__init__::232::IOProcess::(_processLogs) (108905) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:47,655::__init__::232::IOProcess::(_processLogs) (108907) Queuing response >Thread-79::DEBUG::2014-09-25 17:38:47,660::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:38:47,660::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,661::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,661::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,661::__init__::232::IOProcess::(_processLogs) (108908) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:38:47,697::__init__::232::IOProcess::(_processLogs) (108908) Queuing response >Thread-79::DEBUG::2014-09-25 17:38:47,698::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:38:47,699::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,699::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,700::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,700::__init__::232::IOProcess::(_processLogs) (108909) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:38:47,703::__init__::232::IOProcess::(_processLogs) (108909) Queuing response >Thread-79::DEBUG::2014-09-25 17:38:47,704::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:38:47,705::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,705::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,706::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,706::__init__::232::IOProcess::(_processLogs) (108910) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:47,706::__init__::232::IOProcess::(_processLogs) (108910) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:47,707::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,707::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,707::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,708::__init__::232::IOProcess::(_processLogs) (108911) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:38:47,710::__init__::232::IOProcess::(_processLogs) (108911) Queuing response >Thread-79::DEBUG::2014-09-25 17:38:47,711::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 17:38:47,711::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:38:47,711::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:38:47,712::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,712::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,713::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,713::__init__::232::IOProcess::(_processLogs) (108912) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:47,715::__init__::232::IOProcess::(_processLogs) (108912) Queuing response >Thread-79::DEBUG::2014-09-25 17:38:47,716::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:38:47,724::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000440075 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:47,726::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,726::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,726::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,727::__init__::232::IOProcess::(_processLogs) (108913) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:47,727::__init__::232::IOProcess::(_processLogs) (108913) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:47,729::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,729::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,729::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,730::__init__::232::IOProcess::(_processLogs) (108914) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:47,730::__init__::232::IOProcess::(_processLogs) (108914) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:47,731::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,731::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,731::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,732::__init__::232::IOProcess::(_processLogs) (108915) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:47,732::__init__::232::IOProcess::(_processLogs) (108915) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:47,733::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:47,733::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:47,734::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:47,734::__init__::232::IOProcess::(_processLogs) (108916) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:47,734::__init__::232::IOProcess::(_processLogs) (108916) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:48,949::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:48,951::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70794::DEBUG::2014-09-25 17:38:48,953::task::595::Storage.TaskManager.Task::(_updateState) Task=`016fec8e-f972-4154-bb33-c844873c8d72`::moving from state init -> state preparing >Thread-70794::INFO::2014-09-25 17:38:48,953::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70794::INFO::2014-09-25 17:38:48,954::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000440075', 'lastCheck': '1.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000392629', 'lastCheck': '3.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00417369', 'lastCheck': '6.6', 'valid': True}} >Thread-70794::DEBUG::2014-09-25 17:38:48,954::task::1191::Storage.TaskManager.Task::(prepare) Task=`016fec8e-f972-4154-bb33-c844873c8d72`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000440075', 'lastCheck': '1.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000392629', 'lastCheck': '3.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00417369', 'lastCheck': '6.6', 'valid': True}} >Thread-70794::DEBUG::2014-09-25 17:38:48,954::task::595::Storage.TaskManager.Task::(_updateState) Task=`016fec8e-f972-4154-bb33-c844873c8d72`::moving from state preparing -> state finished >Thread-70794::DEBUG::2014-09-25 17:38:48,955::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70794::DEBUG::2014-09-25 17:38:48,955::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70794::DEBUG::2014-09-25 17:38:48,955::task::993::Storage.TaskManager.Task::(_decref) Task=`016fec8e-f972-4154-bb33-c844873c8d72`::ref 0 aborting False >Thread-70794::DEBUG::2014-09-25 17:38:48,957::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:48,978::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:48,979::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70795::DEBUG::2014-09-25 17:38:48,981::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:49,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:49,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0206406 s, 49.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:38:51,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:51,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204972 s, 50.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:52,046::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:52,048::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70796::DEBUG::2014-09-25 17:38:52,050::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:38:52,315::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:52,315::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:52,315::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:52,316::__init__::232::IOProcess::(_processLogs) (108917) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:52,322::__init__::232::IOProcess::(_processLogs) (108917) Queuing response >Thread-81::DEBUG::2014-09-25 17:38:52,323::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:38:52,339::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00306114 s, 121 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:52,340::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:52,341::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:52,341::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:52,341::__init__::232::IOProcess::(_processLogs) (108918) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:52,345::__init__::232::IOProcess::(_processLogs) (108918) Queuing response >Dummy-148::DEBUG::2014-09-25 17:38:53,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:53,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134112 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:55,066::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:55,068::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70797::DEBUG::2014-09-25 17:38:55,070::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:38:55,309::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:55,309::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:55,309::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:55,310::__init__::232::IOProcess::(_processLogs) (108919) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:55,312::__init__::232::IOProcess::(_processLogs) (108919) Queuing response >Thread-80::DEBUG::2014-09-25 17:38:55,313::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:38:55,330::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00406673 s, 92.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:55,331::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:55,331::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:55,332::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:55,332::__init__::232::IOProcess::(_processLogs) (108920) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:55,335::__init__::232::IOProcess::(_processLogs) (108920) Queuing response >Dummy-148::DEBUG::2014-09-25 17:38:55,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:55,424::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184262 s, 55.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:55,900::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:55,901::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70798::DEBUG::2014-09-25 17:38:55,901::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70798::DEBUG::2014-09-25 17:38:55,903::task::595::Storage.TaskManager.Task::(_updateState) Task=`8fc19894-bca3-46be-b360-27d0a1b2d25c`::moving from state init -> state preparing >Thread-70798::INFO::2014-09-25 17:38:55,904::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70798::INFO::2014-09-25 17:38:55,904::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70798::DEBUG::2014-09-25 17:38:55,905::task::1191::Storage.TaskManager.Task::(prepare) Task=`8fc19894-bca3-46be-b360-27d0a1b2d25c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70798::DEBUG::2014-09-25 17:38:55,905::task::595::Storage.TaskManager.Task::(_updateState) Task=`8fc19894-bca3-46be-b360-27d0a1b2d25c`::moving from state preparing -> state finished >Thread-70798::DEBUG::2014-09-25 17:38:55,905::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70798::DEBUG::2014-09-25 17:38:55,905::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70798::DEBUG::2014-09-25 17:38:55,905::task::993::Storage.TaskManager.Task::(_decref) Task=`8fc19894-bca3-46be-b360-27d0a1b2d25c`::ref 0 aborting False >Thread-70798::DEBUG::2014-09-25 17:38:55,906::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70798::DEBUG::2014-09-25 17:38:55,906::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:55,921::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:55,922::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70799::DEBUG::2014-09-25 17:38:55,922::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70799::DEBUG::2014-09-25 17:38:55,924::task::595::Storage.TaskManager.Task::(_updateState) Task=`853c9fea-1c9c-4756-a30d-664f1a513245`::moving from state init -> state preparing >Thread-70799::INFO::2014-09-25 17:38:55,924::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70799::DEBUG::2014-09-25 17:38:55,925::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`18321e67-bf7b-4ef7-aa40-fc9ed3eb6ae7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70799::DEBUG::2014-09-25 17:38:55,925::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70799::DEBUG::2014-09-25 17:38:55,925::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70799::DEBUG::2014-09-25 17:38:55,926::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`18321e67-bf7b-4ef7-aa40-fc9ed3eb6ae7`::Granted request >Thread-70799::DEBUG::2014-09-25 17:38:55,926::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`853c9fea-1c9c-4756-a30d-664f1a513245`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70799::DEBUG::2014-09-25 17:38:55,926::task::993::Storage.TaskManager.Task::(_decref) Task=`853c9fea-1c9c-4756-a30d-664f1a513245`::ref 1 aborting False >Thread-70799::INFO::2014-09-25 17:38:55,931::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70799::DEBUG::2014-09-25 17:38:55,931::task::1191::Storage.TaskManager.Task::(prepare) Task=`853c9fea-1c9c-4756-a30d-664f1a513245`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70799::DEBUG::2014-09-25 17:38:55,931::task::595::Storage.TaskManager.Task::(_updateState) Task=`853c9fea-1c9c-4756-a30d-664f1a513245`::moving from state preparing -> state finished >Thread-70799::DEBUG::2014-09-25 17:38:55,931::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70799::DEBUG::2014-09-25 17:38:55,932::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70799::DEBUG::2014-09-25 17:38:55,932::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70799::DEBUG::2014-09-25 17:38:55,932::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70799::DEBUG::2014-09-25 17:38:55,933::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70799::DEBUG::2014-09-25 17:38:55,933::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70799::DEBUG::2014-09-25 17:38:55,933::task::993::Storage.TaskManager.Task::(_decref) Task=`853c9fea-1c9c-4756-a30d-664f1a513245`::ref 0 aborting False >Thread-70799::DEBUG::2014-09-25 17:38:55,934::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70799::DEBUG::2014-09-25 17:38:55,934::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:57,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:57,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132472 s, 77.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:57,736::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:57,736::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:57,737::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:57,737::__init__::232::IOProcess::(_processLogs) (108921) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:57,737::__init__::232::IOProcess::(_processLogs) (108921) Queuing response >Thread-79::DEBUG::2014-09-25 17:38:57,738::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:38:57,747::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000324917 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:38:57,748::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:57,748::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:57,749::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:57,749::__init__::232::IOProcess::(_processLogs) (108922) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:38:57,749::__init__::232::IOProcess::(_processLogs) (108922) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:57,750::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:57,751::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:57,751::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:57,751::__init__::232::IOProcess::(_processLogs) (108923) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:57,751::__init__::232::IOProcess::(_processLogs) (108923) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:57,752::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:57,752::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:57,753::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:57,754::__init__::232::IOProcess::(_processLogs) (108924) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:57,754::__init__::232::IOProcess::(_processLogs) (108924) Queuing response >Thread-29::DEBUG::2014-09-25 17:38:57,754::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:38:57,755::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:38:57,755::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:38:57,756::__init__::232::IOProcess::(_processLogs) (108925) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:38:57,756::__init__::232::IOProcess::(_processLogs) (108925) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:38:58,090::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:38:58,091::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70800::DEBUG::2014-09-25 17:38:58,093::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:38:59,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:38:59,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0208752 s, 49.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:01,111::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:01,113::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70801::DEBUG::2014-09-25 17:39:01,115::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:01,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:01,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214872 s, 47.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:02,348::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:02,348::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:02,348::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:02,349::__init__::232::IOProcess::(_processLogs) (108926) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:02,361::__init__::232::IOProcess::(_processLogs) (108926) Queuing response >Thread-81::DEBUG::2014-09-25 17:39:02,362::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:39:02,385::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00282338 s, 131 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:02,386::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:02,386::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:02,387::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:02,387::__init__::232::IOProcess::(_processLogs) (108927) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:02,392::__init__::232::IOProcess::(_processLogs) (108927) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:03,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:03,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015629 s, 65.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:04,130::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:04,132::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70802::DEBUG::2014-09-25 17:39:04,135::task::595::Storage.TaskManager.Task::(_updateState) Task=`a72787c1-03c0-4460-8a9a-3fa1e98cebe8`::moving from state init -> state preparing >Thread-70802::INFO::2014-09-25 17:39:04,135::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70802::INFO::2014-09-25 17:39:04,135::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000324917', 'lastCheck': '6.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00406673', 'lastCheck': '8.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00282338', 'lastCheck': '1.7', 'valid': True}} >Thread-70802::DEBUG::2014-09-25 17:39:04,136::task::1191::Storage.TaskManager.Task::(prepare) Task=`a72787c1-03c0-4460-8a9a-3fa1e98cebe8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000324917', 'lastCheck': '6.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00406673', 'lastCheck': '8.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00282338', 'lastCheck': '1.7', 'valid': True}} >Thread-70802::DEBUG::2014-09-25 17:39:04,136::task::595::Storage.TaskManager.Task::(_updateState) Task=`a72787c1-03c0-4460-8a9a-3fa1e98cebe8`::moving from state preparing -> state finished >Thread-70802::DEBUG::2014-09-25 17:39:04,136::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70802::DEBUG::2014-09-25 17:39:04,136::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70802::DEBUG::2014-09-25 17:39:04,136::task::993::Storage.TaskManager.Task::(_decref) Task=`a72787c1-03c0-4460-8a9a-3fa1e98cebe8`::ref 0 aborting False >Thread-70802::DEBUG::2014-09-25 17:39:04,139::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:04,160::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:04,161::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70803::DEBUG::2014-09-25 17:39:04,163::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:39:05,337::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:05,337::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:05,338::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:05,338::__init__::232::IOProcess::(_processLogs) (108928) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:05,338::__init__::232::IOProcess::(_processLogs) (108928) Queuing response >Thread-80::DEBUG::2014-09-25 17:39:05,340::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:39:05,349::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000501435 s, 754 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:05,350::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:05,351::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:05,351::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:05,351::__init__::232::IOProcess::(_processLogs) (108929) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:05,352::__init__::232::IOProcess::(_processLogs) (108929) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:05,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:05,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149302 s, 68.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:05,964::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70804::DEBUG::2014-09-25 17:39:05,965::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:39:05,965::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70804::DEBUG::2014-09-25 17:39:05,969::task::595::Storage.TaskManager.Task::(_updateState) Task=`6441cbd0-f779-4b43-9235-4ad0a5f67f29`::moving from state init -> state preparing >Thread-70804::INFO::2014-09-25 17:39:05,969::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70804::INFO::2014-09-25 17:39:05,970::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70804::DEBUG::2014-09-25 17:39:05,970::task::1191::Storage.TaskManager.Task::(prepare) Task=`6441cbd0-f779-4b43-9235-4ad0a5f67f29`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70804::DEBUG::2014-09-25 17:39:05,970::task::595::Storage.TaskManager.Task::(_updateState) Task=`6441cbd0-f779-4b43-9235-4ad0a5f67f29`::moving from state preparing -> state finished >Thread-70804::DEBUG::2014-09-25 17:39:05,971::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70804::DEBUG::2014-09-25 17:39:05,971::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70804::DEBUG::2014-09-25 17:39:05,971::task::993::Storage.TaskManager.Task::(_decref) Task=`6441cbd0-f779-4b43-9235-4ad0a5f67f29`::ref 0 aborting False >Thread-70804::DEBUG::2014-09-25 17:39:05,971::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70804::DEBUG::2014-09-25 17:39:05,972::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:05,986::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:05,987::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70805::DEBUG::2014-09-25 17:39:05,987::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70805::DEBUG::2014-09-25 17:39:05,989::task::595::Storage.TaskManager.Task::(_updateState) Task=`7f95c725-44a7-4aae-bd37-ad754b39d23c`::moving from state init -> state preparing >Thread-70805::INFO::2014-09-25 17:39:05,989::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70805::DEBUG::2014-09-25 17:39:05,990::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5b44f12b-2fd3-4c68-af6e-923d62d88b4d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70805::DEBUG::2014-09-25 17:39:05,990::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70805::DEBUG::2014-09-25 17:39:05,990::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70805::DEBUG::2014-09-25 17:39:05,991::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`5b44f12b-2fd3-4c68-af6e-923d62d88b4d`::Granted request >Thread-70805::DEBUG::2014-09-25 17:39:05,991::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7f95c725-44a7-4aae-bd37-ad754b39d23c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70805::DEBUG::2014-09-25 17:39:05,992::task::993::Storage.TaskManager.Task::(_decref) Task=`7f95c725-44a7-4aae-bd37-ad754b39d23c`::ref 1 aborting False >Thread-70805::INFO::2014-09-25 17:39:05,996::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70805::DEBUG::2014-09-25 17:39:05,996::task::1191::Storage.TaskManager.Task::(prepare) Task=`7f95c725-44a7-4aae-bd37-ad754b39d23c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70805::DEBUG::2014-09-25 17:39:05,996::task::595::Storage.TaskManager.Task::(_updateState) Task=`7f95c725-44a7-4aae-bd37-ad754b39d23c`::moving from state preparing -> state finished >Thread-70805::DEBUG::2014-09-25 17:39:05,997::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70805::DEBUG::2014-09-25 17:39:05,997::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70805::DEBUG::2014-09-25 17:39:05,997::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70805::DEBUG::2014-09-25 17:39:05,998::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70805::DEBUG::2014-09-25 17:39:05,998::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70805::DEBUG::2014-09-25 17:39:05,998::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70805::DEBUG::2014-09-25 17:39:05,998::task::993::Storage.TaskManager.Task::(_decref) Task=`7f95c725-44a7-4aae-bd37-ad754b39d23c`::ref 0 aborting False >Thread-70805::DEBUG::2014-09-25 17:39:05,999::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70805::DEBUG::2014-09-25 17:39:05,999::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:07,222::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:07,224::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70806::DEBUG::2014-09-25 17:39:07,226::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:07,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:07,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015939 s, 64.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:07,757::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:07,757::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:07,757::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:07,758::__init__::232::IOProcess::(_processLogs) (108930) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:07,760::__init__::232::IOProcess::(_processLogs) (108930) Queuing response >Thread-79::DEBUG::2014-09-25 17:39:07,761::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:39:07,781::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00452335 s, 124 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:07,783::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:07,783::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:07,783::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:07,784::__init__::232::IOProcess::(_processLogs) (108931) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:07,788::__init__::232::IOProcess::(_processLogs) (108931) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:07,790::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:07,790::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:07,790::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:07,791::__init__::232::IOProcess::(_processLogs) (108932) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:07,791::__init__::232::IOProcess::(_processLogs) (108932) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:07,792::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:07,793::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:07,793::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:07,793::__init__::232::IOProcess::(_processLogs) (108933) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:07,793::__init__::232::IOProcess::(_processLogs) (108933) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:07,794::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:07,795::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:07,796::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:07,796::__init__::232::IOProcess::(_processLogs) (108934) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:07,796::__init__::232::IOProcess::(_processLogs) (108934) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:09,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:09,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0238431 s, 42.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:10,243::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:10,245::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70807::DEBUG::2014-09-25 17:39:10,247::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:11,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:11,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0127892 s, 80.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:12,394::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:12,395::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:12,396::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:39:12,396::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:39:12,397::__init__::232::IOProcess::(_processLogs) (108935) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:12,405::__init__::232::IOProcess::(_processLogs) (108935) Queuing response >Thread-81::DEBUG::2014-09-25 17:39:12,406::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000335609 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:12,408::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:12,408::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:12,408::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:12,409::__init__::232::IOProcess::(_processLogs) (108936) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:12,410::__init__::232::IOProcess::(_processLogs) (108936) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:13,264::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:13,266::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70808::DEBUG::2014-09-25 17:39:13,268::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:13,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:13,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171923 s, 59.6 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 17:39:15,353::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:39:15,357::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:15,358::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:15,358::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:15,358::__init__::232::IOProcess::(_processLogs) (108937) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:39:15,359::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:15,360::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:15,360::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:15,361::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:15,361::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:15,361::__init__::232::IOProcess::(_processLogs) (108938) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:39:15,361::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:15,362::__init__::232::IOProcess::(_processLogs) (108939) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:39:15,362::__init__::232::IOProcess::(_processLogs) (108937) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:15,363::__init__::232::IOProcess::(_processLogs) (108938) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:15,366::__init__::232::IOProcess::(_processLogs) (108939) Queuing response >Thread-80::DEBUG::2014-09-25 17:39:15,369::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:39:15,369::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:15,370::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:15,370::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:15,370::__init__::232::IOProcess::(_processLogs) (108940) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:39:15,374::__init__::232::IOProcess::(_processLogs) (108940) Queuing response >Thread-80::DEBUG::2014-09-25 17:39:15,374::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:39:15,376::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:15,376::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:15,376::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:15,376::__init__::232::IOProcess::(_processLogs) (108941) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:39:15,377::__init__::232::IOProcess::(_processLogs) (108941) Queuing response >Thread-80::DEBUG::2014-09-25 17:39:15,377::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:39:15,379::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:15,379::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:15,379::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:15,379::__init__::232::IOProcess::(_processLogs) (108942) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:15,380::__init__::232::IOProcess::(_processLogs) (108942) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:15,381::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:15,381::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:15,381::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:15,382::__init__::232::IOProcess::(_processLogs) (108943) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:39:15,382::__init__::232::IOProcess::(_processLogs) (108943) Queuing response >Thread-80::DEBUG::2014-09-25 17:39:15,382::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-80::INFO::2014-09-25 17:39:15,383::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:39:15,383::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:39:15,384::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:15,384::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:15,384::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:15,385::__init__::232::IOProcess::(_processLogs) (108944) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:15,385::__init__::232::IOProcess::(_processLogs) (108944) Queuing response >Thread-80::DEBUG::2014-09-25 17:39:15,386::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:39:15,395::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000443226 s, 853 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:15,396::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:15,397::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:15,397::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:15,397::__init__::232::IOProcess::(_processLogs) (108945) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:15,398::__init__::232::IOProcess::(_processLogs) (108945) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:15,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:15,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210541 s, 48.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:16,031::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:16,035::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70812::DEBUG::2014-09-25 17:39:16,035::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70812::DEBUG::2014-09-25 17:39:16,041::task::595::Storage.TaskManager.Task::(_updateState) Task=`d4ebb94d-bc7e-4680-b4a7-26df3b767a19`::moving from state init -> state preparing >Thread-70812::INFO::2014-09-25 17:39:16,042::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70812::INFO::2014-09-25 17:39:16,044::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70812::DEBUG::2014-09-25 17:39:16,046::task::1191::Storage.TaskManager.Task::(prepare) Task=`d4ebb94d-bc7e-4680-b4a7-26df3b767a19`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70812::DEBUG::2014-09-25 17:39:16,047::task::595::Storage.TaskManager.Task::(_updateState) Task=`d4ebb94d-bc7e-4680-b4a7-26df3b767a19`::moving from state preparing -> state finished >Thread-70812::DEBUG::2014-09-25 17:39:16,047::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70812::DEBUG::2014-09-25 17:39:16,048::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70812::DEBUG::2014-09-25 17:39:16,050::task::993::Storage.TaskManager.Task::(_decref) Task=`d4ebb94d-bc7e-4680-b4a7-26df3b767a19`::ref 0 aborting False >Thread-70812::DEBUG::2014-09-25 17:39:16,051::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70812::DEBUG::2014-09-25 17:39:16,052::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:16,068::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70813::DEBUG::2014-09-25 17:39:16,071::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:39:16,072::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70813::DEBUG::2014-09-25 17:39:16,076::task::595::Storage.TaskManager.Task::(_updateState) Task=`749957f1-01d0-4600-b278-0e622abf900d`::moving from state init -> state preparing >Thread-70813::INFO::2014-09-25 17:39:16,078::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70813::DEBUG::2014-09-25 17:39:16,081::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4a92be6a-1349-4489-9289-2f6cc0da4d1c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70813::DEBUG::2014-09-25 17:39:16,082::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70813::DEBUG::2014-09-25 17:39:16,083::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70813::DEBUG::2014-09-25 17:39:16,084::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4a92be6a-1349-4489-9289-2f6cc0da4d1c`::Granted request >Thread-70813::DEBUG::2014-09-25 17:39:16,087::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`749957f1-01d0-4600-b278-0e622abf900d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70813::DEBUG::2014-09-25 17:39:16,088::task::993::Storage.TaskManager.Task::(_decref) Task=`749957f1-01d0-4600-b278-0e622abf900d`::ref 1 aborting False >Thread-70813::INFO::2014-09-25 17:39:16,107::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70813::DEBUG::2014-09-25 17:39:16,108::task::1191::Storage.TaskManager.Task::(prepare) Task=`749957f1-01d0-4600-b278-0e622abf900d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70813::DEBUG::2014-09-25 17:39:16,109::task::595::Storage.TaskManager.Task::(_updateState) Task=`749957f1-01d0-4600-b278-0e622abf900d`::moving from state preparing -> state finished >Thread-70813::DEBUG::2014-09-25 17:39:16,110::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70813::DEBUG::2014-09-25 17:39:16,111::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70813::DEBUG::2014-09-25 17:39:16,113::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70813::DEBUG::2014-09-25 17:39:16,114::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70813::DEBUG::2014-09-25 17:39:16,115::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70813::DEBUG::2014-09-25 17:39:16,116::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70813::DEBUG::2014-09-25 17:39:16,118::task::993::Storage.TaskManager.Task::(_decref) Task=`749957f1-01d0-4600-b278-0e622abf900d`::ref 0 aborting False >Thread-70813::DEBUG::2014-09-25 17:39:16,119::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70813::DEBUG::2014-09-25 17:39:16,121::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:16,285::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:16,288::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70814::DEBUG::2014-09-25 17:39:16,293::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:39:17,797::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:17,797::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:17,797::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:17,798::__init__::232::IOProcess::(_processLogs) (108946) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:17,805::__init__::232::IOProcess::(_processLogs) (108946) Queuing response >Thread-79::DEBUG::2014-09-25 17:39:17,806::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:39:17,826::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00717277 s, 78.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:17,827::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:17,828::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:17,828::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:17,828::__init__::232::IOProcess::(_processLogs) (108947) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:17,829::__init__::232::IOProcess::(_processLogs) (108947) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:17,831::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:17,831::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:17,831::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:17,832::__init__::232::IOProcess::(_processLogs) (108948) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:17,832::__init__::232::IOProcess::(_processLogs) (108948) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:17,833::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:17,834::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:17,834::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:17,834::__init__::232::IOProcess::(_processLogs) (108949) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:17,835::__init__::232::IOProcess::(_processLogs) (108949) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:17,835::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:17,836::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:17,837::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:17,837::__init__::232::IOProcess::(_processLogs) (108950) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:17,837::__init__::232::IOProcess::(_processLogs) (108950) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:17,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:17,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145808 s, 70.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:19,309::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:19,311::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70815::DEBUG::2014-09-25 17:39:19,313::task::595::Storage.TaskManager.Task::(_updateState) Task=`b767f241-0e42-437a-9966-5c7c3c627691`::moving from state init -> state preparing >Thread-70815::INFO::2014-09-25 17:39:19,314::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70815::INFO::2014-09-25 17:39:19,314::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00717277', 'lastCheck': '1.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443226', 'lastCheck': '3.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000335609', 'lastCheck': '6.9', 'valid': True}} >Thread-70815::DEBUG::2014-09-25 17:39:19,315::task::1191::Storage.TaskManager.Task::(prepare) Task=`b767f241-0e42-437a-9966-5c7c3c627691`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00717277', 'lastCheck': '1.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443226', 'lastCheck': '3.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000335609', 'lastCheck': '6.9', 'valid': True}} >Thread-70815::DEBUG::2014-09-25 17:39:19,315::task::595::Storage.TaskManager.Task::(_updateState) Task=`b767f241-0e42-437a-9966-5c7c3c627691`::moving from state preparing -> state finished >Thread-70815::DEBUG::2014-09-25 17:39:19,315::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70815::DEBUG::2014-09-25 17:39:19,315::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70815::DEBUG::2014-09-25 17:39:19,315::task::993::Storage.TaskManager.Task::(_decref) Task=`b767f241-0e42-437a-9966-5c7c3c627691`::ref 0 aborting False >Thread-70815::DEBUG::2014-09-25 17:39:19,318::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:19,343::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:19,344::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70816::DEBUG::2014-09-25 17:39:19,346::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:19,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:20,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189931 s, 53.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:39:22,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:22,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015709 s, 65.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:22,411::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:22,412::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:22,412::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:22,412::__init__::232::IOProcess::(_processLogs) (108951) Got request for method 'statvfs' >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:22,413::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-29::DEBUG::2014-09-25 17:39:22,415::__init__::232::IOProcess::(_processLogs) (108951) Queuing response >JsonRpcServer::DEBUG::2014-09-25 17:39:22,416::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-81::DEBUG::2014-09-25 17:39:22,419::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-70817::DEBUG::2014-09-25 17:39:22,429::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 17:39:22,441::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00570838 s, 65.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:22,442::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:22,442::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:22,442::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:22,442::__init__::232::IOProcess::(_processLogs) (108952) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:22,446::__init__::232::IOProcess::(_processLogs) (108952) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:24,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:24,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193824 s, 52.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:25,400::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:25,401::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:25,401::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:25,401::__init__::232::IOProcess::(_processLogs) (108953) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:25,401::__init__::232::IOProcess::(_processLogs) (108953) Queuing response >Thread-80::DEBUG::2014-09-25 17:39:25,403::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:39:25,412::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000492798 s, 767 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:25,413::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:25,413::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:25,413::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:25,414::__init__::232::IOProcess::(_processLogs) (108954) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:25,414::__init__::232::IOProcess::(_processLogs) (108954) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:25,454::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:25,456::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70818::DEBUG::2014-09-25 17:39:25,458::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:26,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:26,152::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:26,155::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70819::DEBUG::2014-09-25 17:39:26,156::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70819::DEBUG::2014-09-25 17:39:26,163::task::595::Storage.TaskManager.Task::(_updateState) Task=`743bdd8b-226e-4ab3-b430-86ea52ee2a6c`::moving from state init -> state preparing >Thread-70819::INFO::2014-09-25 17:39:26,165::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70819::INFO::2014-09-25 17:39:26,169::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70819::DEBUG::2014-09-25 17:39:26,170::task::1191::Storage.TaskManager.Task::(prepare) Task=`743bdd8b-226e-4ab3-b430-86ea52ee2a6c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70819::DEBUG::2014-09-25 17:39:26,172::task::595::Storage.TaskManager.Task::(_updateState) Task=`743bdd8b-226e-4ab3-b430-86ea52ee2a6c`::moving from state preparing -> state finished >Thread-70819::DEBUG::2014-09-25 17:39:26,173::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70819::DEBUG::2014-09-25 17:39:26,175::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70819::DEBUG::2014-09-25 17:39:26,176::task::993::Storage.TaskManager.Task::(_decref) Task=`743bdd8b-226e-4ab3-b430-86ea52ee2a6c`::ref 0 aborting False >Thread-70819::DEBUG::2014-09-25 17:39:26,179::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70819::DEBUG::2014-09-25 17:39:26,181::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:26,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0490115 s, 20.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:26,199::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:26,201::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70820::DEBUG::2014-09-25 17:39:26,202::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70820::DEBUG::2014-09-25 17:39:26,210::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c189eb5-bdb9-42d4-a9bc-68add00a6ac4`::moving from state init -> state preparing >Thread-70820::INFO::2014-09-25 17:39:26,212::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70820::DEBUG::2014-09-25 17:39:26,216::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`eb3a4332-7959-4fa3-9834-b0fb36fc0ccc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70820::DEBUG::2014-09-25 17:39:26,217::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70820::DEBUG::2014-09-25 17:39:26,220::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70820::DEBUG::2014-09-25 17:39:26,221::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`eb3a4332-7959-4fa3-9834-b0fb36fc0ccc`::Granted request >Thread-70820::DEBUG::2014-09-25 17:39:26,225::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`6c189eb5-bdb9-42d4-a9bc-68add00a6ac4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70820::DEBUG::2014-09-25 17:39:26,227::task::993::Storage.TaskManager.Task::(_decref) Task=`6c189eb5-bdb9-42d4-a9bc-68add00a6ac4`::ref 1 aborting False >Thread-70820::INFO::2014-09-25 17:39:26,256::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70820::DEBUG::2014-09-25 17:39:26,258::task::1191::Storage.TaskManager.Task::(prepare) Task=`6c189eb5-bdb9-42d4-a9bc-68add00a6ac4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70820::DEBUG::2014-09-25 17:39:26,260::task::595::Storage.TaskManager.Task::(_updateState) Task=`6c189eb5-bdb9-42d4-a9bc-68add00a6ac4`::moving from state preparing -> state finished >Thread-70820::DEBUG::2014-09-25 17:39:26,261::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70820::DEBUG::2014-09-25 17:39:26,262::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70820::DEBUG::2014-09-25 17:39:26,265::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70820::DEBUG::2014-09-25 17:39:26,267::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70820::DEBUG::2014-09-25 17:39:26,268::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70820::DEBUG::2014-09-25 17:39:26,269::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70820::DEBUG::2014-09-25 17:39:26,272::task::993::Storage.TaskManager.Task::(_decref) Task=`6c189eb5-bdb9-42d4-a9bc-68add00a6ac4`::ref 0 aborting False >Thread-70820::DEBUG::2014-09-25 17:39:26,275::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70820::DEBUG::2014-09-25 17:39:26,278::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:39:27,838::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:27,838::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:27,838::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:27,839::__init__::232::IOProcess::(_processLogs) (108955) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:27,843::__init__::232::IOProcess::(_processLogs) (108955) Queuing response >Thread-79::DEBUG::2014-09-25 17:39:27,844::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:39:27,862::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00555631 s, 101 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:27,864::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:27,864::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:27,864::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:27,864::__init__::232::IOProcess::(_processLogs) (108956) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:27,865::__init__::232::IOProcess::(_processLogs) (108956) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:27,866::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:27,867::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:27,868::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:27,868::__init__::232::IOProcess::(_processLogs) (108957) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:27,868::__init__::232::IOProcess::(_processLogs) (108957) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:27,869::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:27,869::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:27,869::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:27,869::__init__::232::IOProcess::(_processLogs) (108958) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:27,870::__init__::232::IOProcess::(_processLogs) (108958) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:27,870::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:27,871::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:27,871::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:27,871::__init__::232::IOProcess::(_processLogs) (108959) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:27,871::__init__::232::IOProcess::(_processLogs) (108959) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:28,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:28,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143557 s, 71.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:28,474::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:28,475::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70821::DEBUG::2014-09-25 17:39:28,477::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:30,331::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:30,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0219851 s, 46.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:31,492::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:31,494::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70822::DEBUG::2014-09-25 17:39:31,496::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:32,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:32,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133125 s, 76.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:32,448::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:32,448::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:32,448::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:32,448::__init__::232::IOProcess::(_processLogs) (108960) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:32,451::__init__::232::IOProcess::(_processLogs) (108960) Queuing response >Thread-81::DEBUG::2014-09-25 17:39:32,452::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:39:32,467::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0024235 s, 153 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:32,469::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:32,469::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:32,469::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:32,470::__init__::232::IOProcess::(_processLogs) (108961) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:32,476::__init__::232::IOProcess::(_processLogs) (108961) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:34,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:34,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211443 s, 48.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:34,509::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:34,510::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70823::DEBUG::2014-09-25 17:39:34,513::task::595::Storage.TaskManager.Task::(_updateState) Task=`2ddac858-8c8f-449f-a2e3-a492d6b993af`::moving from state init -> state preparing >Thread-70823::INFO::2014-09-25 17:39:34,513::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70823::INFO::2014-09-25 17:39:34,514::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00555631', 'lastCheck': '6.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000492798', 'lastCheck': '9.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0024235', 'lastCheck': '2.0', 'valid': True}} >Thread-70823::DEBUG::2014-09-25 17:39:34,514::task::1191::Storage.TaskManager.Task::(prepare) Task=`2ddac858-8c8f-449f-a2e3-a492d6b993af`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00555631', 'lastCheck': '6.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000492798', 'lastCheck': '9.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0024235', 'lastCheck': '2.0', 'valid': True}} >Thread-70823::DEBUG::2014-09-25 17:39:34,514::task::595::Storage.TaskManager.Task::(_updateState) Task=`2ddac858-8c8f-449f-a2e3-a492d6b993af`::moving from state preparing -> state finished >Thread-70823::DEBUG::2014-09-25 17:39:34,515::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70823::DEBUG::2014-09-25 17:39:34,515::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70823::DEBUG::2014-09-25 17:39:34,515::task::993::Storage.TaskManager.Task::(_decref) Task=`2ddac858-8c8f-449f-a2e3-a492d6b993af`::ref 0 aborting False >Thread-70823::DEBUG::2014-09-25 17:39:34,517::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:34,536::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:34,537::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70824::DEBUG::2014-09-25 17:39:34,539::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:39:35,416::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:35,417::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:35,417::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:35,417::__init__::232::IOProcess::(_processLogs) (108962) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:35,418::__init__::232::IOProcess::(_processLogs) (108962) Queuing response >Thread-80::DEBUG::2014-09-25 17:39:35,419::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:39:35,428::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000442003 s, 855 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:35,430::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:35,430::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:35,430::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:35,430::__init__::232::IOProcess::(_processLogs) (108963) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:35,431::__init__::232::IOProcess::(_processLogs) (108963) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:36,317::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70825::DEBUG::2014-09-25 17:39:36,318::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:39:36,318::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70825::DEBUG::2014-09-25 17:39:36,320::task::595::Storage.TaskManager.Task::(_updateState) Task=`b2a54b67-6fe5-476b-bb15-5f83457861fc`::moving from state init -> state preparing >Thread-70825::INFO::2014-09-25 17:39:36,321::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70825::INFO::2014-09-25 17:39:36,321::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70825::DEBUG::2014-09-25 17:39:36,321::task::1191::Storage.TaskManager.Task::(prepare) Task=`b2a54b67-6fe5-476b-bb15-5f83457861fc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70825::DEBUG::2014-09-25 17:39:36,322::task::595::Storage.TaskManager.Task::(_updateState) Task=`b2a54b67-6fe5-476b-bb15-5f83457861fc`::moving from state preparing -> state finished >Thread-70825::DEBUG::2014-09-25 17:39:36,322::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70825::DEBUG::2014-09-25 17:39:36,322::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70825::DEBUG::2014-09-25 17:39:36,322::task::993::Storage.TaskManager.Task::(_decref) Task=`b2a54b67-6fe5-476b-bb15-5f83457861fc`::ref 0 aborting False >Thread-70825::DEBUG::2014-09-25 17:39:36,323::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70825::DEBUG::2014-09-25 17:39:36,323::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:36,336::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:36,337::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70826::DEBUG::2014-09-25 17:39:36,337::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70826::DEBUG::2014-09-25 17:39:36,339::task::595::Storage.TaskManager.Task::(_updateState) Task=`28a9232d-ffc1-461c-89f1-22fe60648983`::moving from state init -> state preparing >Thread-70826::INFO::2014-09-25 17:39:36,339::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70826::DEBUG::2014-09-25 17:39:36,340::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ed0f459e-da9f-4be7-be2c-ee352f580702`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70826::DEBUG::2014-09-25 17:39:36,340::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70826::DEBUG::2014-09-25 17:39:36,341::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70826::DEBUG::2014-09-25 17:39:36,341::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ed0f459e-da9f-4be7-be2c-ee352f580702`::Granted request >Thread-70826::DEBUG::2014-09-25 17:39:36,342::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`28a9232d-ffc1-461c-89f1-22fe60648983`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70826::DEBUG::2014-09-25 17:39:36,342::task::993::Storage.TaskManager.Task::(_decref) Task=`28a9232d-ffc1-461c-89f1-22fe60648983`::ref 1 aborting False >Thread-70826::INFO::2014-09-25 17:39:36,346::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70826::DEBUG::2014-09-25 17:39:36,346::task::1191::Storage.TaskManager.Task::(prepare) Task=`28a9232d-ffc1-461c-89f1-22fe60648983`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70826::DEBUG::2014-09-25 17:39:36,347::task::595::Storage.TaskManager.Task::(_updateState) Task=`28a9232d-ffc1-461c-89f1-22fe60648983`::moving from state preparing -> state finished >Thread-70826::DEBUG::2014-09-25 17:39:36,347::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70826::DEBUG::2014-09-25 17:39:36,347::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70826::DEBUG::2014-09-25 17:39:36,347::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70826::DEBUG::2014-09-25 17:39:36,348::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70826::DEBUG::2014-09-25 17:39:36,348::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70826::DEBUG::2014-09-25 17:39:36,348::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70826::DEBUG::2014-09-25 17:39:36,349::task::993::Storage.TaskManager.Task::(_decref) Task=`28a9232d-ffc1-461c-89f1-22fe60648983`::ref 0 aborting False >Thread-70826::DEBUG::2014-09-25 17:39:36,349::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70826::DEBUG::2014-09-25 17:39:36,349::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:36,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:36,511::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163919 s, 62.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:37,602::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:37,603::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70827::DEBUG::2014-09-25 17:39:37,605::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:39:37,873::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:37,873::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:37,873::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:37,873::__init__::232::IOProcess::(_processLogs) (108964) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:37,874::__init__::232::IOProcess::(_processLogs) (108964) Queuing response >Thread-79::DEBUG::2014-09-25 17:39:37,875::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:39:37,884::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000377587 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:37,885::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:37,885::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:37,885::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:37,886::__init__::232::IOProcess::(_processLogs) (108965) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:37,886::__init__::232::IOProcess::(_processLogs) (108965) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:37,887::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:37,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:37,889::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:37,889::__init__::232::IOProcess::(_processLogs) (108966) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:37,889::__init__::232::IOProcess::(_processLogs) (108966) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:37,889::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:37,890::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:37,890::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:37,891::__init__::232::IOProcess::(_processLogs) (108967) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:37,891::__init__::232::IOProcess::(_processLogs) (108967) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:37,891::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:37,892::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:37,893::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:37,893::__init__::232::IOProcess::(_processLogs) (108968) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:37,893::__init__::232::IOProcess::(_processLogs) (108968) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:38,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:38,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0261165 s, 39.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:39:40,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:40,622::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:40,626::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70828::DEBUG::2014-09-25 17:39:40,631::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:40,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0251782 s, 40.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:42,478::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:42,478::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:42,479::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:42,479::__init__::232::IOProcess::(_processLogs) (108969) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:42,482::__init__::232::IOProcess::(_processLogs) (108969) Queuing response >Thread-81::DEBUG::2014-09-25 17:39:42,483::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:39:42,499::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00179894 s, 206 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:42,500::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:42,500::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:42,500::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:42,501::__init__::232::IOProcess::(_processLogs) (108970) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:42,506::__init__::232::IOProcess::(_processLogs) (108970) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:42,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:42,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135456 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:43,648::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:43,650::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70829::DEBUG::2014-09-25 17:39:43,652::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:44,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:44,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0187748 s, 54.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:45,433::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:45,434::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:45,434::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:45,434::__init__::232::IOProcess::(_processLogs) (108971) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:45,435::__init__::232::IOProcess::(_processLogs) (108971) Queuing response >Thread-80::DEBUG::2014-09-25 17:39:45,436::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:39:45,445::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000362708 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:45,447::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:45,447::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:45,447::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:45,448::__init__::232::IOProcess::(_processLogs) (108972) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:45,448::__init__::232::IOProcess::(_processLogs) (108972) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:46,378::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70830::DEBUG::2014-09-25 17:39:46,382::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:39:46,382::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70830::DEBUG::2014-09-25 17:39:46,387::task::595::Storage.TaskManager.Task::(_updateState) Task=`4345209a-f7c0-4f23-8715-b3b86992bfb1`::moving from state init -> state preparing >Thread-70830::INFO::2014-09-25 17:39:46,388::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70830::INFO::2014-09-25 17:39:46,390::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70830::DEBUG::2014-09-25 17:39:46,391::task::1191::Storage.TaskManager.Task::(prepare) Task=`4345209a-f7c0-4f23-8715-b3b86992bfb1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70830::DEBUG::2014-09-25 17:39:46,392::task::595::Storage.TaskManager.Task::(_updateState) Task=`4345209a-f7c0-4f23-8715-b3b86992bfb1`::moving from state preparing -> state finished >Thread-70830::DEBUG::2014-09-25 17:39:46,392::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70830::DEBUG::2014-09-25 17:39:46,393::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70830::DEBUG::2014-09-25 17:39:46,394::task::993::Storage.TaskManager.Task::(_decref) Task=`4345209a-f7c0-4f23-8715-b3b86992bfb1`::ref 0 aborting False >Thread-70830::DEBUG::2014-09-25 17:39:46,395::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70830::DEBUG::2014-09-25 17:39:46,397::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:46,413::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70831::DEBUG::2014-09-25 17:39:46,417::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:39:46,417::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70831::DEBUG::2014-09-25 17:39:46,421::task::595::Storage.TaskManager.Task::(_updateState) Task=`bad06ee0-effb-4097-9986-8df1955faeab`::moving from state init -> state preparing >Thread-70831::INFO::2014-09-25 17:39:46,422::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70831::DEBUG::2014-09-25 17:39:46,425::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7c255cc4-d53a-4e9f-8e9e-55773dca1e29`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70831::DEBUG::2014-09-25 17:39:46,425::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70831::DEBUG::2014-09-25 17:39:46,427::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70831::DEBUG::2014-09-25 17:39:46,428::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7c255cc4-d53a-4e9f-8e9e-55773dca1e29`::Granted request >Thread-70831::DEBUG::2014-09-25 17:39:46,431::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`bad06ee0-effb-4097-9986-8df1955faeab`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70831::DEBUG::2014-09-25 17:39:46,432::task::993::Storage.TaskManager.Task::(_decref) Task=`bad06ee0-effb-4097-9986-8df1955faeab`::ref 1 aborting False >Thread-70831::INFO::2014-09-25 17:39:46,449::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70831::DEBUG::2014-09-25 17:39:46,450::task::1191::Storage.TaskManager.Task::(prepare) Task=`bad06ee0-effb-4097-9986-8df1955faeab`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70831::DEBUG::2014-09-25 17:39:46,451::task::595::Storage.TaskManager.Task::(_updateState) Task=`bad06ee0-effb-4097-9986-8df1955faeab`::moving from state preparing -> state finished >Thread-70831::DEBUG::2014-09-25 17:39:46,451::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70831::DEBUG::2014-09-25 17:39:46,452::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70831::DEBUG::2014-09-25 17:39:46,454::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70831::DEBUG::2014-09-25 17:39:46,455::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70831::DEBUG::2014-09-25 17:39:46,456::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70831::DEBUG::2014-09-25 17:39:46,456::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70831::DEBUG::2014-09-25 17:39:46,458::task::993::Storage.TaskManager.Task::(_decref) Task=`bad06ee0-effb-4097-9986-8df1955faeab`::ref 0 aborting False >Thread-70831::DEBUG::2014-09-25 17:39:46,460::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70831::DEBUG::2014-09-25 17:39:46,461::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:46,667::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:46,670::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70832::DEBUG::2014-09-25 17:39:46,674::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:46,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:46,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0350045 s, 29.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:47,894::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:47,894::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:47,894::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:47,895::__init__::232::IOProcess::(_processLogs) (108973) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:47,897::__init__::232::IOProcess::(_processLogs) (108973) Queuing response >Thread-79::DEBUG::2014-09-25 17:39:47,898::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:39:47,911::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00231183 s, 243 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:47,912::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:47,912::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:47,912::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:47,913::__init__::232::IOProcess::(_processLogs) (108974) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:47,917::__init__::232::IOProcess::(_processLogs) (108974) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:47,918::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:47,919::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:47,919::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:47,919::__init__::232::IOProcess::(_processLogs) (108975) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:47,924::__init__::232::IOProcess::(_processLogs) (108975) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:47,925::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:47,925::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:47,926::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:47,926::__init__::232::IOProcess::(_processLogs) (108976) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:47,933::__init__::232::IOProcess::(_processLogs) (108976) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:47,934::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:47,934::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:47,934::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:47,934::__init__::232::IOProcess::(_processLogs) (108977) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:47,938::__init__::232::IOProcess::(_processLogs) (108977) Queuing response >Dummy-148::DEBUG::2014-09-25 17:39:48,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:49,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174991 s, 58.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:49,691::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:49,693::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70833::DEBUG::2014-09-25 17:39:49,695::task::595::Storage.TaskManager.Task::(_updateState) Task=`71090e89-49da-4091-9fc3-4da12e184eb9`::moving from state init -> state preparing >Thread-70833::INFO::2014-09-25 17:39:49,696::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70833::INFO::2014-09-25 17:39:49,696::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00231183', 'lastCheck': '1.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000362708', 'lastCheck': '4.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00179894', 'lastCheck': '7.2', 'valid': True}} >Thread-70833::DEBUG::2014-09-25 17:39:49,696::task::1191::Storage.TaskManager.Task::(prepare) Task=`71090e89-49da-4091-9fc3-4da12e184eb9`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00231183', 'lastCheck': '1.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000362708', 'lastCheck': '4.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00179894', 'lastCheck': '7.2', 'valid': True}} >Thread-70833::DEBUG::2014-09-25 17:39:49,697::task::595::Storage.TaskManager.Task::(_updateState) Task=`71090e89-49da-4091-9fc3-4da12e184eb9`::moving from state preparing -> state finished >Thread-70833::DEBUG::2014-09-25 17:39:49,697::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70833::DEBUG::2014-09-25 17:39:49,697::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70833::DEBUG::2014-09-25 17:39:49,697::task::993::Storage.TaskManager.Task::(_decref) Task=`71090e89-49da-4091-9fc3-4da12e184eb9`::ref 0 aborting False >Thread-70833::DEBUG::2014-09-25 17:39:49,700::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:49,720::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:49,721::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70834::DEBUG::2014-09-25 17:39:49,723::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:51,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:51,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167905 s, 61.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:52,508::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:52,509::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:52,509::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:52,509::__init__::232::IOProcess::(_processLogs) (108978) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:52,512::__init__::232::IOProcess::(_processLogs) (108978) Queuing response >Thread-81::DEBUG::2014-09-25 17:39:52,513::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:39:52,528::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00385804 s, 96.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:52,529::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:52,529::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:52,530::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:52,530::__init__::232::IOProcess::(_processLogs) (108979) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:52,533::__init__::232::IOProcess::(_processLogs) (108979) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:52,780::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:52,781::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70835::DEBUG::2014-09-25 17:39:52,783::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:53,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:53,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175985 s, 58.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:39:55,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:55,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133676 s, 76.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:55,451::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:55,451::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:55,451::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:55,451::__init__::232::IOProcess::(_processLogs) (108980) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:55,452::__init__::232::IOProcess::(_processLogs) (108980) Queuing response >Thread-80::DEBUG::2014-09-25 17:39:55,453::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:39:55,462::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000484412 s, 780 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:55,463::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:55,463::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:55,463::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:55,464::__init__::232::IOProcess::(_processLogs) (108981) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:55,464::__init__::232::IOProcess::(_processLogs) (108981) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:55,799::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:55,801::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70836::DEBUG::2014-09-25 17:39:55,802::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:56,493::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:56,494::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70837::DEBUG::2014-09-25 17:39:56,494::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70837::DEBUG::2014-09-25 17:39:56,496::task::595::Storage.TaskManager.Task::(_updateState) Task=`2dea4df6-9d31-421b-a5e5-7d814fb1720c`::moving from state init -> state preparing >Thread-70837::INFO::2014-09-25 17:39:56,496::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70837::INFO::2014-09-25 17:39:56,497::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70837::DEBUG::2014-09-25 17:39:56,497::task::1191::Storage.TaskManager.Task::(prepare) Task=`2dea4df6-9d31-421b-a5e5-7d814fb1720c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70837::DEBUG::2014-09-25 17:39:56,498::task::595::Storage.TaskManager.Task::(_updateState) Task=`2dea4df6-9d31-421b-a5e5-7d814fb1720c`::moving from state preparing -> state finished >Thread-70837::DEBUG::2014-09-25 17:39:56,498::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70837::DEBUG::2014-09-25 17:39:56,498::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70837::DEBUG::2014-09-25 17:39:56,498::task::993::Storage.TaskManager.Task::(_decref) Task=`2dea4df6-9d31-421b-a5e5-7d814fb1720c`::ref 0 aborting False >Thread-70837::DEBUG::2014-09-25 17:39:56,498::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70837::DEBUG::2014-09-25 17:39:56,499::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:56,531::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:56,532::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70838::DEBUG::2014-09-25 17:39:56,532::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70838::DEBUG::2014-09-25 17:39:56,534::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc34bdf4-23a0-422f-8298-57eec60e79f1`::moving from state init -> state preparing >Thread-70838::INFO::2014-09-25 17:39:56,534::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70838::DEBUG::2014-09-25 17:39:56,535::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1787db59-a602-41ae-bbb8-75cba2f2cb2e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70838::DEBUG::2014-09-25 17:39:56,535::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70838::DEBUG::2014-09-25 17:39:56,536::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70838::DEBUG::2014-09-25 17:39:56,536::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1787db59-a602-41ae-bbb8-75cba2f2cb2e`::Granted request >Thread-70838::DEBUG::2014-09-25 17:39:56,537::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`dc34bdf4-23a0-422f-8298-57eec60e79f1`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70838::DEBUG::2014-09-25 17:39:56,537::task::993::Storage.TaskManager.Task::(_decref) Task=`dc34bdf4-23a0-422f-8298-57eec60e79f1`::ref 1 aborting False >Thread-70838::INFO::2014-09-25 17:39:56,541::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70838::DEBUG::2014-09-25 17:39:56,541::task::1191::Storage.TaskManager.Task::(prepare) Task=`dc34bdf4-23a0-422f-8298-57eec60e79f1`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70838::DEBUG::2014-09-25 17:39:56,542::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc34bdf4-23a0-422f-8298-57eec60e79f1`::moving from state preparing -> state finished >Thread-70838::DEBUG::2014-09-25 17:39:56,542::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70838::DEBUG::2014-09-25 17:39:56,542::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70838::DEBUG::2014-09-25 17:39:56,542::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70838::DEBUG::2014-09-25 17:39:56,543::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70838::DEBUG::2014-09-25 17:39:56,543::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70838::DEBUG::2014-09-25 17:39:56,543::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70838::DEBUG::2014-09-25 17:39:56,544::task::993::Storage.TaskManager.Task::(_decref) Task=`dc34bdf4-23a0-422f-8298-57eec60e79f1`::ref 0 aborting False >Thread-70838::DEBUG::2014-09-25 17:39:56,544::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70838::DEBUG::2014-09-25 17:39:56,544::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:57,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:57,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210267 s, 48.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:57,940::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:57,941::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:57,941::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:57,941::__init__::232::IOProcess::(_processLogs) (108982) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:57,946::__init__::232::IOProcess::(_processLogs) (108982) Queuing response >Thread-79::DEBUG::2014-09-25 17:39:57,947::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:39:57,968::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00507462 s, 111 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:39:57,969::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:57,970::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:57,970::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:57,970::__init__::232::IOProcess::(_processLogs) (108983) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:39:57,974::__init__::232::IOProcess::(_processLogs) (108983) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:57,975::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:57,975::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:57,976::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:57,977::__init__::232::IOProcess::(_processLogs) (108984) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:57,977::__init__::232::IOProcess::(_processLogs) (108984) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:57,977::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:57,978::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:57,978::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:57,979::__init__::232::IOProcess::(_processLogs) (108985) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:57,979::__init__::232::IOProcess::(_processLogs) (108985) Queuing response >Thread-29::DEBUG::2014-09-25 17:39:57,979::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:39:57,980::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:39:57,980::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:39:57,980::__init__::232::IOProcess::(_processLogs) (108986) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:39:57,980::__init__::232::IOProcess::(_processLogs) (108986) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:39:58,820::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:39:58,821::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70839::DEBUG::2014-09-25 17:39:58,823::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:39:59,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:39:59,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135418 s, 75.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:40:01,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:01,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016924 s, 60.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:01,846::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:01,848::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70840::DEBUG::2014-09-25 17:40:01,850::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:40:02,535::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:02,535::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:02,536::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:02,536::__init__::232::IOProcess::(_processLogs) (108987) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:02,549::__init__::232::IOProcess::(_processLogs) (108987) Queuing response >Thread-81::DEBUG::2014-09-25 17:40:02,550::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:40:02,576::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00667307 s, 55.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:02,577::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:02,577::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:02,577::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:02,578::__init__::232::IOProcess::(_processLogs) (108988) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:02,585::__init__::232::IOProcess::(_processLogs) (108988) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:03,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:03,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131291 s, 78.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:04,864::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:04,866::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70841::DEBUG::2014-09-25 17:40:04,868::task::595::Storage.TaskManager.Task::(_updateState) Task=`caec9113-3dbe-4bde-80c6-ae7a294b49a5`::moving from state init -> state preparing >Thread-70841::INFO::2014-09-25 17:40:04,869::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70841::INFO::2014-09-25 17:40:04,869::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00507462', 'lastCheck': '6.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000484412', 'lastCheck': '9.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00667307', 'lastCheck': '2.3', 'valid': True}} >Thread-70841::DEBUG::2014-09-25 17:40:04,869::task::1191::Storage.TaskManager.Task::(prepare) Task=`caec9113-3dbe-4bde-80c6-ae7a294b49a5`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00507462', 'lastCheck': '6.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000484412', 'lastCheck': '9.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00667307', 'lastCheck': '2.3', 'valid': True}} >Thread-70841::DEBUG::2014-09-25 17:40:04,870::task::595::Storage.TaskManager.Task::(_updateState) Task=`caec9113-3dbe-4bde-80c6-ae7a294b49a5`::moving from state preparing -> state finished >Thread-70841::DEBUG::2014-09-25 17:40:04,870::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70841::DEBUG::2014-09-25 17:40:04,870::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70841::DEBUG::2014-09-25 17:40:04,870::task::993::Storage.TaskManager.Task::(_decref) Task=`caec9113-3dbe-4bde-80c6-ae7a294b49a5`::ref 0 aborting False >Thread-70841::DEBUG::2014-09-25 17:40:04,873::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:04,898::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:04,899::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70842::DEBUG::2014-09-25 17:40:04,901::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:05,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:05,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204256 s, 50.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:05,466::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:05,467::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:05,467::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:05,467::__init__::232::IOProcess::(_processLogs) (108989) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:05,468::__init__::232::IOProcess::(_processLogs) (108989) Queuing response >Thread-80::DEBUG::2014-09-25 17:40:05,469::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:40:05,478::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000357309 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:05,479::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:05,479::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:05,480::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:05,481::__init__::232::IOProcess::(_processLogs) (108990) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:05,482::__init__::232::IOProcess::(_processLogs) (108990) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:06,575::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70843::DEBUG::2014-09-25 17:40:06,576::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:40:06,576::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70843::DEBUG::2014-09-25 17:40:06,578::task::595::Storage.TaskManager.Task::(_updateState) Task=`705271fa-385d-452a-873c-708693a5eec5`::moving from state init -> state preparing >Thread-70843::INFO::2014-09-25 17:40:06,579::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70843::INFO::2014-09-25 17:40:06,579::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70843::DEBUG::2014-09-25 17:40:06,580::task::1191::Storage.TaskManager.Task::(prepare) Task=`705271fa-385d-452a-873c-708693a5eec5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70843::DEBUG::2014-09-25 17:40:06,580::task::595::Storage.TaskManager.Task::(_updateState) Task=`705271fa-385d-452a-873c-708693a5eec5`::moving from state preparing -> state finished >Thread-70843::DEBUG::2014-09-25 17:40:06,580::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70843::DEBUG::2014-09-25 17:40:06,580::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70843::DEBUG::2014-09-25 17:40:06,580::task::993::Storage.TaskManager.Task::(_decref) Task=`705271fa-385d-452a-873c-708693a5eec5`::ref 0 aborting False >Thread-70843::DEBUG::2014-09-25 17:40:06,581::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70843::DEBUG::2014-09-25 17:40:06,581::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:06,606::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:06,607::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70844::DEBUG::2014-09-25 17:40:06,607::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70844::DEBUG::2014-09-25 17:40:06,608::task::595::Storage.TaskManager.Task::(_updateState) Task=`19eebf42-2fec-4a38-8f4c-bb1553d12fe3`::moving from state init -> state preparing >Thread-70844::INFO::2014-09-25 17:40:06,609::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70844::DEBUG::2014-09-25 17:40:06,609::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`883c7515-0b48-4fe7-8ed5-94a211767444`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70844::DEBUG::2014-09-25 17:40:06,610::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70844::DEBUG::2014-09-25 17:40:06,610::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70844::DEBUG::2014-09-25 17:40:06,610::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`883c7515-0b48-4fe7-8ed5-94a211767444`::Granted request >Thread-70844::DEBUG::2014-09-25 17:40:06,611::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`19eebf42-2fec-4a38-8f4c-bb1553d12fe3`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70844::DEBUG::2014-09-25 17:40:06,611::task::993::Storage.TaskManager.Task::(_decref) Task=`19eebf42-2fec-4a38-8f4c-bb1553d12fe3`::ref 1 aborting False >Thread-70844::INFO::2014-09-25 17:40:06,615::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70844::DEBUG::2014-09-25 17:40:06,616::task::1191::Storage.TaskManager.Task::(prepare) Task=`19eebf42-2fec-4a38-8f4c-bb1553d12fe3`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70844::DEBUG::2014-09-25 17:40:06,616::task::595::Storage.TaskManager.Task::(_updateState) Task=`19eebf42-2fec-4a38-8f4c-bb1553d12fe3`::moving from state preparing -> state finished >Thread-70844::DEBUG::2014-09-25 17:40:06,616::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70844::DEBUG::2014-09-25 17:40:06,616::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70844::DEBUG::2014-09-25 17:40:06,617::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70844::DEBUG::2014-09-25 17:40:06,617::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70844::DEBUG::2014-09-25 17:40:06,617::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70844::DEBUG::2014-09-25 17:40:06,618::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70844::DEBUG::2014-09-25 17:40:06,618::task::993::Storage.TaskManager.Task::(_decref) Task=`19eebf42-2fec-4a38-8f4c-bb1553d12fe3`::ref 0 aborting False >Thread-70844::DEBUG::2014-09-25 17:40:06,618::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70844::DEBUG::2014-09-25 17:40:06,619::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:07,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:07,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161945 s, 63.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:07,968::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:07,969::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70845::DEBUG::2014-09-25 17:40:07,971::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:40:07,982::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:07,982::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:07,982::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:07,982::__init__::232::IOProcess::(_processLogs) (108991) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:07,986::__init__::232::IOProcess::(_processLogs) (108991) Queuing response >Thread-79::DEBUG::2014-09-25 17:40:07,987::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:40:08,005::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00277971 s, 202 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:08,006::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:08,007::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:08,007::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:08,007::__init__::232::IOProcess::(_processLogs) (108992) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:08,011::__init__::232::IOProcess::(_processLogs) (108992) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:08,013::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:08,013::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:08,014::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:08,015::__init__::232::IOProcess::(_processLogs) (108993) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:08,015::__init__::232::IOProcess::(_processLogs) (108993) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:08,015::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:08,015::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:08,016::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:08,016::__init__::232::IOProcess::(_processLogs) (108994) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:08,016::__init__::232::IOProcess::(_processLogs) (108994) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:08,017::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:08,017::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:08,018::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:08,019::__init__::232::IOProcess::(_processLogs) (108995) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:08,019::__init__::232::IOProcess::(_processLogs) (108995) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:09,492::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:09,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168189 s, 60.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:10,983::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:10,985::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70846::DEBUG::2014-09-25 17:40:10,987::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:11,538::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:11,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152705 s, 67.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:12,587::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:12,588::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:12,588::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:12,588::__init__::232::IOProcess::(_processLogs) (108996) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:12,595::__init__::232::IOProcess::(_processLogs) (108996) Queuing response >Thread-81::DEBUG::2014-09-25 17:40:12,596::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:40:12,613::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00538143 s, 68.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:12,614::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:12,614::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:12,614::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:12,615::__init__::232::IOProcess::(_processLogs) (108997) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:12,616::__init__::232::IOProcess::(_processLogs) (108997) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:13,593::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:13,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017651 s, 58.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:13,999::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:14,001::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70847::DEBUG::2014-09-25 17:40:14,003::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:40:15,482::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:15,483::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:15,483::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:15,483::__init__::232::IOProcess::(_processLogs) (108998) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:15,484::__init__::232::IOProcess::(_processLogs) (108998) Queuing response >Thread-80::DEBUG::2014-09-25 17:40:15,485::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:40:15,494::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000428286 s, 883 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:15,495::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:15,495::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:15,495::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:15,496::__init__::232::IOProcess::(_processLogs) (108999) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:15,496::__init__::232::IOProcess::(_processLogs) (108999) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:15,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:15,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016633 s, 61.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:16,648::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70848::DEBUG::2014-09-25 17:40:16,652::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:40:16,652::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70848::DEBUG::2014-09-25 17:40:16,658::task::595::Storage.TaskManager.Task::(_updateState) Task=`deae0588-07bc-4556-9022-a3f46d6cc188`::moving from state init -> state preparing >Thread-70848::INFO::2014-09-25 17:40:16,659::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70848::INFO::2014-09-25 17:40:16,660::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70848::DEBUG::2014-09-25 17:40:16,662::task::1191::Storage.TaskManager.Task::(prepare) Task=`deae0588-07bc-4556-9022-a3f46d6cc188`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70848::DEBUG::2014-09-25 17:40:16,662::task::595::Storage.TaskManager.Task::(_updateState) Task=`deae0588-07bc-4556-9022-a3f46d6cc188`::moving from state preparing -> state finished >Thread-70848::DEBUG::2014-09-25 17:40:16,663::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70848::DEBUG::2014-09-25 17:40:16,664::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70848::DEBUG::2014-09-25 17:40:16,665::task::993::Storage.TaskManager.Task::(_decref) Task=`deae0588-07bc-4556-9022-a3f46d6cc188`::ref 0 aborting False >Thread-70848::DEBUG::2014-09-25 17:40:16,666::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70848::DEBUG::2014-09-25 17:40:16,667::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:16,680::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:16,683::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70849::DEBUG::2014-09-25 17:40:16,683::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70849::DEBUG::2014-09-25 17:40:16,687::task::595::Storage.TaskManager.Task::(_updateState) Task=`15d2a087-bd51-4628-b840-6a7016c8621b`::moving from state init -> state preparing >Thread-70849::INFO::2014-09-25 17:40:16,688::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70849::DEBUG::2014-09-25 17:40:16,691::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6a6a4fb8-3957-41a3-9d61-d0413aadd945`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70849::DEBUG::2014-09-25 17:40:16,692::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70849::DEBUG::2014-09-25 17:40:16,693::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70849::DEBUG::2014-09-25 17:40:16,694::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6a6a4fb8-3957-41a3-9d61-d0413aadd945`::Granted request >Thread-70849::DEBUG::2014-09-25 17:40:16,696::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`15d2a087-bd51-4628-b840-6a7016c8621b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70849::DEBUG::2014-09-25 17:40:16,697::task::993::Storage.TaskManager.Task::(_decref) Task=`15d2a087-bd51-4628-b840-6a7016c8621b`::ref 1 aborting False >Thread-70849::INFO::2014-09-25 17:40:16,714::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70849::DEBUG::2014-09-25 17:40:16,715::task::1191::Storage.TaskManager.Task::(prepare) Task=`15d2a087-bd51-4628-b840-6a7016c8621b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70849::DEBUG::2014-09-25 17:40:16,716::task::595::Storage.TaskManager.Task::(_updateState) Task=`15d2a087-bd51-4628-b840-6a7016c8621b`::moving from state preparing -> state finished >Thread-70849::DEBUG::2014-09-25 17:40:16,717::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70849::DEBUG::2014-09-25 17:40:16,718::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70849::DEBUG::2014-09-25 17:40:16,719::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70849::DEBUG::2014-09-25 17:40:16,720::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70849::DEBUG::2014-09-25 17:40:16,721::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70849::DEBUG::2014-09-25 17:40:16,722::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70849::DEBUG::2014-09-25 17:40:16,725::task::993::Storage.TaskManager.Task::(_decref) Task=`15d2a087-bd51-4628-b840-6a7016c8621b`::ref 0 aborting False >Thread-70849::DEBUG::2014-09-25 17:40:16,729::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70849::DEBUG::2014-09-25 17:40:16,731::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:17,017::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:17,017::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70850::DEBUG::2014-09-25 17:40:17,019::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:17,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:17,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0217108 s, 47.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:18,019::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:18,020::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:18,020::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:18,020::__init__::232::IOProcess::(_processLogs) (109000) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:18,033::__init__::232::IOProcess::(_processLogs) (109000) Queuing response >Thread-79::DEBUG::2014-09-25 17:40:18,034::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:40:18,056::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0082979 s, 67.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:18,057::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:18,058::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:18,058::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:18,058::__init__::232::IOProcess::(_processLogs) (109001) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:18,061::__init__::232::IOProcess::(_processLogs) (109001) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:18,062::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:18,063::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:18,063::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:18,063::__init__::232::IOProcess::(_processLogs) (109002) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:18,063::__init__::232::IOProcess::(_processLogs) (109002) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:18,064::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:18,065::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:18,065::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:18,066::__init__::232::IOProcess::(_processLogs) (109003) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:18,066::__init__::232::IOProcess::(_processLogs) (109003) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:18,066::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:18,067::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:18,068::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:18,068::__init__::232::IOProcess::(_processLogs) (109004) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:18,068::__init__::232::IOProcess::(_processLogs) (109004) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:19,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:19,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184737 s, 55.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:20,034::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:20,036::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70851::DEBUG::2014-09-25 17:40:20,038::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d5381c7-a386-4881-a790-127de0f6f37f`::moving from state init -> state preparing >Thread-70851::INFO::2014-09-25 17:40:20,039::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70851::INFO::2014-09-25 17:40:20,039::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0082979', 'lastCheck': '2.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000428286', 'lastCheck': '4.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00538143', 'lastCheck': '7.4', 'valid': True}} >Thread-70851::DEBUG::2014-09-25 17:40:20,039::task::1191::Storage.TaskManager.Task::(prepare) Task=`6d5381c7-a386-4881-a790-127de0f6f37f`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0082979', 'lastCheck': '2.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000428286', 'lastCheck': '4.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00538143', 'lastCheck': '7.4', 'valid': True}} >Thread-70851::DEBUG::2014-09-25 17:40:20,040::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d5381c7-a386-4881-a790-127de0f6f37f`::moving from state preparing -> state finished >Thread-70851::DEBUG::2014-09-25 17:40:20,040::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70851::DEBUG::2014-09-25 17:40:20,040::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70851::DEBUG::2014-09-25 17:40:20,040::task::993::Storage.TaskManager.Task::(_decref) Task=`6d5381c7-a386-4881-a790-127de0f6f37f`::ref 0 aborting False >Thread-70851::DEBUG::2014-09-25 17:40:20,043::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:20,062::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:20,063::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70852::DEBUG::2014-09-25 17:40:20,065::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:21,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:21,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188172 s, 54.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:22,618::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:22,618::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:22,619::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:22,619::__init__::232::IOProcess::(_processLogs) (109005) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:22,624::__init__::232::IOProcess::(_processLogs) (109005) Queuing response >Thread-81::DEBUG::2014-09-25 17:40:22,626::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:40:22,654::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0049069 s, 75.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:22,655::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:22,656::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:22,656::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:22,656::__init__::232::IOProcess::(_processLogs) (109006) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:22,662::__init__::232::IOProcess::(_processLogs) (109006) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:23,158::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:23,159::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70853::DEBUG::2014-09-25 17:40:23,161::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:23,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:23,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134088 s, 76.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:25,499::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:25,499::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:25,499::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:25,500::__init__::232::IOProcess::(_processLogs) (109007) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:25,500::__init__::232::IOProcess::(_processLogs) (109007) Queuing response >Thread-80::DEBUG::2014-09-25 17:40:25,501::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:40:25,511::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000521066 s, 725 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:25,512::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:25,512::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:25,513::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:25,514::__init__::232::IOProcess::(_processLogs) (109008) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:25,515::__init__::232::IOProcess::(_processLogs) (109008) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:25,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:25,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138437 s, 74.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:26,177::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:26,179::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70854::DEBUG::2014-09-25 17:40:26,181::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:26,760::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:26,761::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70855::DEBUG::2014-09-25 17:40:26,761::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70855::DEBUG::2014-09-25 17:40:26,763::task::595::Storage.TaskManager.Task::(_updateState) Task=`71514b33-2390-47ba-a5ad-c352108e0e29`::moving from state init -> state preparing >Thread-70855::INFO::2014-09-25 17:40:26,763::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70855::INFO::2014-09-25 17:40:26,763::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70855::DEBUG::2014-09-25 17:40:26,764::task::1191::Storage.TaskManager.Task::(prepare) Task=`71514b33-2390-47ba-a5ad-c352108e0e29`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70855::DEBUG::2014-09-25 17:40:26,764::task::595::Storage.TaskManager.Task::(_updateState) Task=`71514b33-2390-47ba-a5ad-c352108e0e29`::moving from state preparing -> state finished >Thread-70855::DEBUG::2014-09-25 17:40:26,764::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70855::DEBUG::2014-09-25 17:40:26,764::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70855::DEBUG::2014-09-25 17:40:26,765::task::993::Storage.TaskManager.Task::(_decref) Task=`71514b33-2390-47ba-a5ad-c352108e0e29`::ref 0 aborting False >Thread-70855::DEBUG::2014-09-25 17:40:26,765::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70855::DEBUG::2014-09-25 17:40:26,765::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:26,780::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:26,783::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70856::DEBUG::2014-09-25 17:40:26,783::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70856::DEBUG::2014-09-25 17:40:26,787::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd6fc789-9d3c-4f14-9e1e-efb8c65159e0`::moving from state init -> state preparing >Thread-70856::INFO::2014-09-25 17:40:26,788::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70856::DEBUG::2014-09-25 17:40:26,790::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3898864c-3d3b-41e0-859a-f357357eacee`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70856::DEBUG::2014-09-25 17:40:26,792::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70856::DEBUG::2014-09-25 17:40:26,794::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70856::DEBUG::2014-09-25 17:40:26,794::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3898864c-3d3b-41e0-859a-f357357eacee`::Granted request >Thread-70856::DEBUG::2014-09-25 17:40:26,796::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`dd6fc789-9d3c-4f14-9e1e-efb8c65159e0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70856::DEBUG::2014-09-25 17:40:26,796::task::993::Storage.TaskManager.Task::(_decref) Task=`dd6fc789-9d3c-4f14-9e1e-efb8c65159e0`::ref 1 aborting False >Thread-70856::INFO::2014-09-25 17:40:26,800::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70856::DEBUG::2014-09-25 17:40:26,800::task::1191::Storage.TaskManager.Task::(prepare) Task=`dd6fc789-9d3c-4f14-9e1e-efb8c65159e0`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70856::DEBUG::2014-09-25 17:40:26,801::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd6fc789-9d3c-4f14-9e1e-efb8c65159e0`::moving from state preparing -> state finished >Thread-70856::DEBUG::2014-09-25 17:40:26,801::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70856::DEBUG::2014-09-25 17:40:26,801::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70856::DEBUG::2014-09-25 17:40:26,802::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70856::DEBUG::2014-09-25 17:40:26,802::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70856::DEBUG::2014-09-25 17:40:26,802::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70856::DEBUG::2014-09-25 17:40:26,802::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70856::DEBUG::2014-09-25 17:40:26,803::task::993::Storage.TaskManager.Task::(_decref) Task=`dd6fc789-9d3c-4f14-9e1e-efb8c65159e0`::ref 0 aborting False >Thread-70856::DEBUG::2014-09-25 17:40:26,803::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70856::DEBUG::2014-09-25 17:40:26,804::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:27,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:27,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170993 s, 59.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:28,069::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:28,069::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:28,069::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:28,069::__init__::232::IOProcess::(_processLogs) (109009) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:28,070::__init__::232::IOProcess::(_processLogs) (109009) Queuing response >Thread-79::DEBUG::2014-09-25 17:40:28,071::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:40:28,087::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0018399 s, 305 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:28,088::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:28,089::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:28,089::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:28,089::__init__::232::IOProcess::(_processLogs) (109010) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:28,094::__init__::232::IOProcess::(_processLogs) (109010) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:28,096::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:28,096::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:28,097::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:28,097::__init__::232::IOProcess::(_processLogs) (109011) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:28,097::__init__::232::IOProcess::(_processLogs) (109011) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:28,098::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:28,098::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:28,099::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:28,099::__init__::232::IOProcess::(_processLogs) (109012) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:28,100::__init__::232::IOProcess::(_processLogs) (109012) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:28,100::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:28,100::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:28,101::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:28,102::__init__::232::IOProcess::(_processLogs) (109013) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:28,102::__init__::232::IOProcess::(_processLogs) (109013) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:29,199::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:29,201::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70857::DEBUG::2014-09-25 17:40:29,203::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:30,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:30,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158308 s, 64.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:40:32,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:32,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171783 s, 59.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:32,220::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:32,221::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70858::DEBUG::2014-09-25 17:40:32,223::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:40:32,665::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:32,665::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:32,666::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:32,667::__init__::232::IOProcess::(_processLogs) (109014) Got request for method 'statvfs' >Thread-81::DEBUG::2014-09-25 17:40:32,667::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:40:32,669::__init__::232::IOProcess::(_processLogs) (109014) Queuing response >Thread-81::DEBUG::2014-09-25 17:40:32,677::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000274323 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:32,678::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:32,678::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:32,678::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:32,679::__init__::232::IOProcess::(_processLogs) (109015) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:32,679::__init__::232::IOProcess::(_processLogs) (109015) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:34,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:34,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151358 s, 67.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:35,240::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:35,241::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70859::DEBUG::2014-09-25 17:40:35,244::task::595::Storage.TaskManager.Task::(_updateState) Task=`d221f1c6-7279-45ca-a402-712f620535a1`::moving from state init -> state preparing >Thread-70859::INFO::2014-09-25 17:40:35,244::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70859::INFO::2014-09-25 17:40:35,245::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0018399', 'lastCheck': '7.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000521066', 'lastCheck': '9.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000274323', 'lastCheck': '2.6', 'valid': True}} >Thread-70859::DEBUG::2014-09-25 17:40:35,245::task::1191::Storage.TaskManager.Task::(prepare) Task=`d221f1c6-7279-45ca-a402-712f620535a1`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0018399', 'lastCheck': '7.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000521066', 'lastCheck': '9.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000274323', 'lastCheck': '2.6', 'valid': True}} >Thread-70859::DEBUG::2014-09-25 17:40:35,245::task::595::Storage.TaskManager.Task::(_updateState) Task=`d221f1c6-7279-45ca-a402-712f620535a1`::moving from state preparing -> state finished >Thread-70859::DEBUG::2014-09-25 17:40:35,245::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70859::DEBUG::2014-09-25 17:40:35,246::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70859::DEBUG::2014-09-25 17:40:35,246::task::993::Storage.TaskManager.Task::(_decref) Task=`d221f1c6-7279-45ca-a402-712f620535a1`::ref 0 aborting False >Thread-70859::DEBUG::2014-09-25 17:40:35,248::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:35,269::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:35,270::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70860::DEBUG::2014-09-25 17:40:35,272::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:40:35,515::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:35,515::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:35,516::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:35,516::__init__::232::IOProcess::(_processLogs) (109016) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:35,516::__init__::232::IOProcess::(_processLogs) (109016) Queuing response >Thread-80::DEBUG::2014-09-25 17:40:35,518::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:40:35,527::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000543517 s, 695 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:35,528::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:35,528::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:35,528::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:35,529::__init__::232::IOProcess::(_processLogs) (109017) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:35,530::__init__::232::IOProcess::(_processLogs) (109017) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:36,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:36,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0246536 s, 41.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:36,832::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70861::DEBUG::2014-09-25 17:40:36,833::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:40:36,833::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70861::DEBUG::2014-09-25 17:40:36,836::task::595::Storage.TaskManager.Task::(_updateState) Task=`87ce747b-cbac-4e85-a8b1-4e2f269254e0`::moving from state init -> state preparing >Thread-70861::INFO::2014-09-25 17:40:36,836::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70861::INFO::2014-09-25 17:40:36,837::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70861::DEBUG::2014-09-25 17:40:36,837::task::1191::Storage.TaskManager.Task::(prepare) Task=`87ce747b-cbac-4e85-a8b1-4e2f269254e0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70861::DEBUG::2014-09-25 17:40:36,837::task::595::Storage.TaskManager.Task::(_updateState) Task=`87ce747b-cbac-4e85-a8b1-4e2f269254e0`::moving from state preparing -> state finished >Thread-70861::DEBUG::2014-09-25 17:40:36,837::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70861::DEBUG::2014-09-25 17:40:36,838::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70861::DEBUG::2014-09-25 17:40:36,838::task::993::Storage.TaskManager.Task::(_decref) Task=`87ce747b-cbac-4e85-a8b1-4e2f269254e0`::ref 0 aborting False >Thread-70861::DEBUG::2014-09-25 17:40:36,838::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70861::DEBUG::2014-09-25 17:40:36,838::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:36,852::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70862::DEBUG::2014-09-25 17:40:36,853::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:40:36,853::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70862::DEBUG::2014-09-25 17:40:36,855::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c570f73-0822-4cc8-9dcf-51de7bf55fd7`::moving from state init -> state preparing >Thread-70862::INFO::2014-09-25 17:40:36,856::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70862::DEBUG::2014-09-25 17:40:36,856::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8f50c758-2b3d-430e-b4a6-1143ad4cc9a8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70862::DEBUG::2014-09-25 17:40:36,857::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70862::DEBUG::2014-09-25 17:40:36,857::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70862::DEBUG::2014-09-25 17:40:36,857::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8f50c758-2b3d-430e-b4a6-1143ad4cc9a8`::Granted request >Thread-70862::DEBUG::2014-09-25 17:40:36,858::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2c570f73-0822-4cc8-9dcf-51de7bf55fd7`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70862::DEBUG::2014-09-25 17:40:36,858::task::993::Storage.TaskManager.Task::(_decref) Task=`2c570f73-0822-4cc8-9dcf-51de7bf55fd7`::ref 1 aborting False >Thread-70862::INFO::2014-09-25 17:40:36,862::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70862::DEBUG::2014-09-25 17:40:36,863::task::1191::Storage.TaskManager.Task::(prepare) Task=`2c570f73-0822-4cc8-9dcf-51de7bf55fd7`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70862::DEBUG::2014-09-25 17:40:36,863::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c570f73-0822-4cc8-9dcf-51de7bf55fd7`::moving from state preparing -> state finished >Thread-70862::DEBUG::2014-09-25 17:40:36,863::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70862::DEBUG::2014-09-25 17:40:36,863::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70862::DEBUG::2014-09-25 17:40:36,864::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70862::DEBUG::2014-09-25 17:40:36,864::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70862::DEBUG::2014-09-25 17:40:36,864::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70862::DEBUG::2014-09-25 17:40:36,865::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70862::DEBUG::2014-09-25 17:40:36,865::task::993::Storage.TaskManager.Task::(_decref) Task=`2c570f73-0822-4cc8-9dcf-51de7bf55fd7`::ref 0 aborting False >Thread-70862::DEBUG::2014-09-25 17:40:36,865::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70862::DEBUG::2014-09-25 17:40:36,866::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:40:38,103::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:38,103::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:38,103::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:38,103::__init__::232::IOProcess::(_processLogs) (109018) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:38,111::__init__::232::IOProcess::(_processLogs) (109018) Queuing response >Thread-79::DEBUG::2014-09-25 17:40:38,112::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:40:38,134::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0038707 s, 145 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:38,135::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:38,135::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:38,136::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:38,136::__init__::232::IOProcess::(_processLogs) (109019) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:38,142::__init__::232::IOProcess::(_processLogs) (109019) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:38,143::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:38,143::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:38,144::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:38,144::__init__::232::IOProcess::(_processLogs) (109020) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:38,145::__init__::232::IOProcess::(_processLogs) (109020) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:38,145::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:38,146::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:38,146::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:38,146::__init__::232::IOProcess::(_processLogs) (109021) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:38,147::__init__::232::IOProcess::(_processLogs) (109021) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:38,147::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:38,147::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:38,148::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:38,149::__init__::232::IOProcess::(_processLogs) (109022) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:38,149::__init__::232::IOProcess::(_processLogs) (109022) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:38,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:38,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.025228 s, 40.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:38,331::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:38,332::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70863::DEBUG::2014-09-25 17:40:38,335::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:40,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:40,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0319682 s, 32.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:41,350::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:41,351::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70864::DEBUG::2014-09-25 17:40:41,353::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:42,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:42,452::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132028 s, 77.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:42,682::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:42,683::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:42,683::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:42,683::__init__::232::IOProcess::(_processLogs) (109023) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:42,686::__init__::232::IOProcess::(_processLogs) (109023) Queuing response >Thread-81::DEBUG::2014-09-25 17:40:42,687::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:40:42,696::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000352564 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:42,697::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:42,697::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:42,698::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:42,698::__init__::232::IOProcess::(_processLogs) (109024) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:42,699::__init__::232::IOProcess::(_processLogs) (109024) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:44,369::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:44,370::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70865::DEBUG::2014-09-25 17:40:44,372::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:44,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:44,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192813 s, 53.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:45,531::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:45,532::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:45,532::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:45,532::__init__::232::IOProcess::(_processLogs) (109025) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:45,533::__init__::232::IOProcess::(_processLogs) (109025) Queuing response >Thread-80::DEBUG::2014-09-25 17:40:45,534::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:40:45,544::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000352136 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:45,545::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:45,546::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:45,546::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:45,546::__init__::232::IOProcess::(_processLogs) (109026) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:45,547::__init__::232::IOProcess::(_processLogs) (109026) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:46,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:46,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0240149 s, 42.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:46,897::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:46,901::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70866::DEBUG::2014-09-25 17:40:46,902::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70866::DEBUG::2014-09-25 17:40:46,907::task::595::Storage.TaskManager.Task::(_updateState) Task=`21aeddbc-ed6f-444e-b35b-a03b2cbde65a`::moving from state init -> state preparing >Thread-70866::INFO::2014-09-25 17:40:46,908::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70866::INFO::2014-09-25 17:40:46,910::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70866::DEBUG::2014-09-25 17:40:46,911::task::1191::Storage.TaskManager.Task::(prepare) Task=`21aeddbc-ed6f-444e-b35b-a03b2cbde65a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70866::DEBUG::2014-09-25 17:40:46,912::task::595::Storage.TaskManager.Task::(_updateState) Task=`21aeddbc-ed6f-444e-b35b-a03b2cbde65a`::moving from state preparing -> state finished >Thread-70866::DEBUG::2014-09-25 17:40:46,913::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70866::DEBUG::2014-09-25 17:40:46,914::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70866::DEBUG::2014-09-25 17:40:46,914::task::993::Storage.TaskManager.Task::(_decref) Task=`21aeddbc-ed6f-444e-b35b-a03b2cbde65a`::ref 0 aborting False >Thread-70866::DEBUG::2014-09-25 17:40:46,916::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70866::DEBUG::2014-09-25 17:40:46,917::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:46,933::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:46,937::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70867::DEBUG::2014-09-25 17:40:46,937::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70867::DEBUG::2014-09-25 17:40:46,941::task::595::Storage.TaskManager.Task::(_updateState) Task=`60eeb2a4-c6c4-495c-814e-bb0c1501fe92`::moving from state init -> state preparing >Thread-70867::INFO::2014-09-25 17:40:46,942::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70867::DEBUG::2014-09-25 17:40:46,945::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8e047ee0-c9cc-4185-905b-379d09b824a8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70867::DEBUG::2014-09-25 17:40:46,946::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70867::DEBUG::2014-09-25 17:40:46,947::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70867::DEBUG::2014-09-25 17:40:46,948::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8e047ee0-c9cc-4185-905b-379d09b824a8`::Granted request >Thread-70867::DEBUG::2014-09-25 17:40:46,950::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`60eeb2a4-c6c4-495c-814e-bb0c1501fe92`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70867::DEBUG::2014-09-25 17:40:46,951::task::993::Storage.TaskManager.Task::(_decref) Task=`60eeb2a4-c6c4-495c-814e-bb0c1501fe92`::ref 1 aborting False >Thread-70867::INFO::2014-09-25 17:40:46,968::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70867::DEBUG::2014-09-25 17:40:46,969::task::1191::Storage.TaskManager.Task::(prepare) Task=`60eeb2a4-c6c4-495c-814e-bb0c1501fe92`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70867::DEBUG::2014-09-25 17:40:46,970::task::595::Storage.TaskManager.Task::(_updateState) Task=`60eeb2a4-c6c4-495c-814e-bb0c1501fe92`::moving from state preparing -> state finished >Thread-70867::DEBUG::2014-09-25 17:40:46,970::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70867::DEBUG::2014-09-25 17:40:46,971::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70867::DEBUG::2014-09-25 17:40:46,973::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70867::DEBUG::2014-09-25 17:40:46,974::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70867::DEBUG::2014-09-25 17:40:46,975::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70867::DEBUG::2014-09-25 17:40:46,975::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70867::DEBUG::2014-09-25 17:40:46,977::task::993::Storage.TaskManager.Task::(_decref) Task=`60eeb2a4-c6c4-495c-814e-bb0c1501fe92`::ref 0 aborting False >Thread-70867::DEBUG::2014-09-25 17:40:46,978::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70867::DEBUG::2014-09-25 17:40:46,980::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:47,390::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:47,392::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70868::DEBUG::2014-09-25 17:40:47,394::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:40:48,150::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:48,150::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:48,150::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:48,151::__init__::232::IOProcess::(_processLogs) (109027) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:48,163::__init__::232::IOProcess::(_processLogs) (109027) Queuing response >Thread-79::DEBUG::2014-09-25 17:40:48,164::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:40:48,185::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00494035 s, 114 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:48,186::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:48,187::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:48,187::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:48,187::__init__::232::IOProcess::(_processLogs) (109028) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:48,190::__init__::232::IOProcess::(_processLogs) (109028) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:48,191::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:48,191::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:48,191::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:48,192::__init__::232::IOProcess::(_processLogs) (109029) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:48,193::__init__::232::IOProcess::(_processLogs) (109029) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:48,194::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:48,195::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:48,195::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:48,195::__init__::232::IOProcess::(_processLogs) (109030) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:48,197::__init__::232::IOProcess::(_processLogs) (109030) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:48,198::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:48,199::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:48,199::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:48,199::__init__::232::IOProcess::(_processLogs) (109031) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:48,202::__init__::232::IOProcess::(_processLogs) (109031) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:48,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:48,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135895 s, 75.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:50,410::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:50,411::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70869::DEBUG::2014-09-25 17:40:50,414::task::595::Storage.TaskManager.Task::(_updateState) Task=`05ae9a09-147c-4cd0-b1bb-f96d3fcc86f8`::moving from state init -> state preparing >Thread-70869::INFO::2014-09-25 17:40:50,414::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70869::INFO::2014-09-25 17:40:50,415::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00494035', 'lastCheck': '2.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000352136', 'lastCheck': '4.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000352564', 'lastCheck': '7.7', 'valid': True}} >Thread-70869::DEBUG::2014-09-25 17:40:50,415::task::1191::Storage.TaskManager.Task::(prepare) Task=`05ae9a09-147c-4cd0-b1bb-f96d3fcc86f8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00494035', 'lastCheck': '2.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000352136', 'lastCheck': '4.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000352564', 'lastCheck': '7.7', 'valid': True}} >Thread-70869::DEBUG::2014-09-25 17:40:50,415::task::595::Storage.TaskManager.Task::(_updateState) Task=`05ae9a09-147c-4cd0-b1bb-f96d3fcc86f8`::moving from state preparing -> state finished >Thread-70869::DEBUG::2014-09-25 17:40:50,416::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70869::DEBUG::2014-09-25 17:40:50,416::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70869::DEBUG::2014-09-25 17:40:50,416::task::993::Storage.TaskManager.Task::(_decref) Task=`05ae9a09-147c-4cd0-b1bb-f96d3fcc86f8`::ref 0 aborting False >Thread-70869::DEBUG::2014-09-25 17:40:50,418::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:50,445::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:50,448::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70870::DEBUG::2014-09-25 17:40:50,455::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:50,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:50,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0227598 s, 45.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:52,701::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:52,701::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:52,701::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:52,702::__init__::232::IOProcess::(_processLogs) (109032) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:52,704::__init__::232::IOProcess::(_processLogs) (109032) Queuing response >Thread-81::DEBUG::2014-09-25 17:40:52,705::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:40:52,717::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00157356 s, 236 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:52,718::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:52,718::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:52,718::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:52,719::__init__::232::IOProcess::(_processLogs) (109033) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:52,721::__init__::232::IOProcess::(_processLogs) (109033) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:52,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:52,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136017 s, 75.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:53,517::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:53,518::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70871::DEBUG::2014-09-25 17:40:53,520::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:54,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:54,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142416 s, 71.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:55,549::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:55,549::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:55,550::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:55,550::__init__::232::IOProcess::(_processLogs) (109034) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:55,550::__init__::232::IOProcess::(_processLogs) (109034) Queuing response >Thread-80::DEBUG::2014-09-25 17:40:55,551::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:40:55,560::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000536146 s, 705 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:55,561::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:55,562::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:55,562::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:55,562::__init__::232::IOProcess::(_processLogs) (109035) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:55,563::__init__::232::IOProcess::(_processLogs) (109035) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:56,538::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:56,539::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70872::DEBUG::2014-09-25 17:40:56,541::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:40:56,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:56,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158042 s, 64.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:57,010::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70873::DEBUG::2014-09-25 17:40:57,012::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:40:57,012::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70873::DEBUG::2014-09-25 17:40:57,015::task::595::Storage.TaskManager.Task::(_updateState) Task=`e0c20476-93ca-494b-b523-a005dabcef39`::moving from state init -> state preparing >Thread-70873::INFO::2014-09-25 17:40:57,015::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70873::INFO::2014-09-25 17:40:57,015::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70873::DEBUG::2014-09-25 17:40:57,016::task::1191::Storage.TaskManager.Task::(prepare) Task=`e0c20476-93ca-494b-b523-a005dabcef39`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70873::DEBUG::2014-09-25 17:40:57,016::task::595::Storage.TaskManager.Task::(_updateState) Task=`e0c20476-93ca-494b-b523-a005dabcef39`::moving from state preparing -> state finished >Thread-70873::DEBUG::2014-09-25 17:40:57,016::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70873::DEBUG::2014-09-25 17:40:57,016::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70873::DEBUG::2014-09-25 17:40:57,017::task::993::Storage.TaskManager.Task::(_decref) Task=`e0c20476-93ca-494b-b523-a005dabcef39`::ref 0 aborting False >Thread-70873::DEBUG::2014-09-25 17:40:57,017::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70873::DEBUG::2014-09-25 17:40:57,017::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:57,031::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70874::DEBUG::2014-09-25 17:40:57,032::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:40:57,032::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70874::DEBUG::2014-09-25 17:40:57,034::task::595::Storage.TaskManager.Task::(_updateState) Task=`96bf9490-d611-4c6d-80cf-2284127f8b9a`::moving from state init -> state preparing >Thread-70874::INFO::2014-09-25 17:40:57,034::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70874::DEBUG::2014-09-25 17:40:57,035::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`969691aa-741a-4fce-81a5-e76a0bfa1bb4`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70874::DEBUG::2014-09-25 17:40:57,035::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70874::DEBUG::2014-09-25 17:40:57,036::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70874::DEBUG::2014-09-25 17:40:57,036::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`969691aa-741a-4fce-81a5-e76a0bfa1bb4`::Granted request >Thread-70874::DEBUG::2014-09-25 17:40:57,036::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`96bf9490-d611-4c6d-80cf-2284127f8b9a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70874::DEBUG::2014-09-25 17:40:57,037::task::993::Storage.TaskManager.Task::(_decref) Task=`96bf9490-d611-4c6d-80cf-2284127f8b9a`::ref 1 aborting False >Thread-70874::INFO::2014-09-25 17:40:57,041::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70874::DEBUG::2014-09-25 17:40:57,041::task::1191::Storage.TaskManager.Task::(prepare) Task=`96bf9490-d611-4c6d-80cf-2284127f8b9a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70874::DEBUG::2014-09-25 17:40:57,042::task::595::Storage.TaskManager.Task::(_updateState) Task=`96bf9490-d611-4c6d-80cf-2284127f8b9a`::moving from state preparing -> state finished >Thread-70874::DEBUG::2014-09-25 17:40:57,042::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70874::DEBUG::2014-09-25 17:40:57,042::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70874::DEBUG::2014-09-25 17:40:57,042::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70874::DEBUG::2014-09-25 17:40:57,043::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70874::DEBUG::2014-09-25 17:40:57,043::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70874::DEBUG::2014-09-25 17:40:57,043::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70874::DEBUG::2014-09-25 17:40:57,044::task::993::Storage.TaskManager.Task::(_decref) Task=`96bf9490-d611-4c6d-80cf-2284127f8b9a`::ref 0 aborting False >Thread-70874::DEBUG::2014-09-25 17:40:57,044::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70874::DEBUG::2014-09-25 17:40:57,044::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:40:58,203::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:58,204::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:58,204::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:58,204::__init__::232::IOProcess::(_processLogs) (109036) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:58,211::__init__::232::IOProcess::(_processLogs) (109036) Queuing response >Thread-79::DEBUG::2014-09-25 17:40:58,212::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:40:58,242::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0095073 s, 59.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:40:58,244::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:58,244::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:58,244::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:58,244::__init__::232::IOProcess::(_processLogs) (109037) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:40:58,250::__init__::232::IOProcess::(_processLogs) (109037) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:58,252::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:58,252::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:58,253::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:58,253::__init__::232::IOProcess::(_processLogs) (109038) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:58,253::__init__::232::IOProcess::(_processLogs) (109038) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:58,254::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:58,254::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:58,254::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:58,254::__init__::232::IOProcess::(_processLogs) (109039) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:58,255::__init__::232::IOProcess::(_processLogs) (109039) Queuing response >Thread-29::DEBUG::2014-09-25 17:40:58,256::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:40:58,256::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:40:58,256::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:40:58,257::__init__::232::IOProcess::(_processLogs) (109040) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:40:58,257::__init__::232::IOProcess::(_processLogs) (109040) Queuing response >Dummy-148::DEBUG::2014-09-25 17:40:58,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:40:59,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203844 s, 50.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:40:59,559::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:40:59,560::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70875::DEBUG::2014-09-25 17:40:59,562::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:01,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:01,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017279 s, 59.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:02,578::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:02,580::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70876::DEBUG::2014-09-25 17:41:02,582::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:41:02,724::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:02,724::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:02,724::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:02,724::__init__::232::IOProcess::(_processLogs) (109041) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:02,736::__init__::232::IOProcess::(_processLogs) (109041) Queuing response >Thread-81::DEBUG::2014-09-25 17:41:02,737::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:41:02,760::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00369398 s, 100 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:02,761::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:02,761::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:02,761::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:02,762::__init__::232::IOProcess::(_processLogs) (109042) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:02,766::__init__::232::IOProcess::(_processLogs) (109042) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:03,093::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:03,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157047 s, 65.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:41:05,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:05,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146382 s, 70.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:05,565::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:05,566::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:05,566::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:05,566::__init__::232::IOProcess::(_processLogs) (109043) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:05,567::__init__::232::IOProcess::(_processLogs) (109043) Queuing response >Thread-80::DEBUG::2014-09-25 17:41:05,568::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:41:05,577::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000522675 s, 723 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:05,578::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:05,578::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:05,579::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:05,579::__init__::232::IOProcess::(_processLogs) (109044) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:05,580::__init__::232::IOProcess::(_processLogs) (109044) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:05,596::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:05,598::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70877::DEBUG::2014-09-25 17:41:05,600::task::595::Storage.TaskManager.Task::(_updateState) Task=`6aa1b2fe-dd34-4ff8-94d1-d4443979d751`::moving from state init -> state preparing >Thread-70877::INFO::2014-09-25 17:41:05,601::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70877::INFO::2014-09-25 17:41:05,601::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0095073', 'lastCheck': '7.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000522675', 'lastCheck': '0.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00369398', 'lastCheck': '2.8', 'valid': True}} >Thread-70877::DEBUG::2014-09-25 17:41:05,601::task::1191::Storage.TaskManager.Task::(prepare) Task=`6aa1b2fe-dd34-4ff8-94d1-d4443979d751`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0095073', 'lastCheck': '7.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000522675', 'lastCheck': '0.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00369398', 'lastCheck': '2.8', 'valid': True}} >Thread-70877::DEBUG::2014-09-25 17:41:05,602::task::595::Storage.TaskManager.Task::(_updateState) Task=`6aa1b2fe-dd34-4ff8-94d1-d4443979d751`::moving from state preparing -> state finished >Thread-70877::DEBUG::2014-09-25 17:41:05,602::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70877::DEBUG::2014-09-25 17:41:05,602::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70877::DEBUG::2014-09-25 17:41:05,602::task::993::Storage.TaskManager.Task::(_decref) Task=`6aa1b2fe-dd34-4ff8-94d1-d4443979d751`::ref 0 aborting False >Thread-70877::DEBUG::2014-09-25 17:41:05,605::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:05,625::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:05,626::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70878::DEBUG::2014-09-25 17:41:05,628::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:07,073::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70879::DEBUG::2014-09-25 17:41:07,074::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:41:07,074::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70879::DEBUG::2014-09-25 17:41:07,077::task::595::Storage.TaskManager.Task::(_updateState) Task=`b3fc3db4-65e0-4018-bd9f-3263eaf5a421`::moving from state init -> state preparing >Thread-70879::INFO::2014-09-25 17:41:07,077::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70879::INFO::2014-09-25 17:41:07,077::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70879::DEBUG::2014-09-25 17:41:07,078::task::1191::Storage.TaskManager.Task::(prepare) Task=`b3fc3db4-65e0-4018-bd9f-3263eaf5a421`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70879::DEBUG::2014-09-25 17:41:07,078::task::595::Storage.TaskManager.Task::(_updateState) Task=`b3fc3db4-65e0-4018-bd9f-3263eaf5a421`::moving from state preparing -> state finished >Thread-70879::DEBUG::2014-09-25 17:41:07,078::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70879::DEBUG::2014-09-25 17:41:07,078::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70879::DEBUG::2014-09-25 17:41:07,078::task::993::Storage.TaskManager.Task::(_decref) Task=`b3fc3db4-65e0-4018-bd9f-3263eaf5a421`::ref 0 aborting False >Thread-70879::DEBUG::2014-09-25 17:41:07,079::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70879::DEBUG::2014-09-25 17:41:07,079::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:07,093::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:07,094::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70880::DEBUG::2014-09-25 17:41:07,095::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70880::DEBUG::2014-09-25 17:41:07,096::task::595::Storage.TaskManager.Task::(_updateState) Task=`09f4be80-ce86-4f1e-a1b5-aa5555380c81`::moving from state init -> state preparing >Thread-70880::INFO::2014-09-25 17:41:07,096::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70880::DEBUG::2014-09-25 17:41:07,097::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`63735383-6717-4848-af47-27f7256868a5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70880::DEBUG::2014-09-25 17:41:07,097::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70880::DEBUG::2014-09-25 17:41:07,098::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70880::DEBUG::2014-09-25 17:41:07,098::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`63735383-6717-4848-af47-27f7256868a5`::Granted request >Thread-70880::DEBUG::2014-09-25 17:41:07,099::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`09f4be80-ce86-4f1e-a1b5-aa5555380c81`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70880::DEBUG::2014-09-25 17:41:07,099::task::993::Storage.TaskManager.Task::(_decref) Task=`09f4be80-ce86-4f1e-a1b5-aa5555380c81`::ref 1 aborting False >Thread-70880::INFO::2014-09-25 17:41:07,103::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70880::DEBUG::2014-09-25 17:41:07,103::task::1191::Storage.TaskManager.Task::(prepare) Task=`09f4be80-ce86-4f1e-a1b5-aa5555380c81`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70880::DEBUG::2014-09-25 17:41:07,104::task::595::Storage.TaskManager.Task::(_updateState) Task=`09f4be80-ce86-4f1e-a1b5-aa5555380c81`::moving from state preparing -> state finished >Thread-70880::DEBUG::2014-09-25 17:41:07,104::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70880::DEBUG::2014-09-25 17:41:07,104::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70880::DEBUG::2014-09-25 17:41:07,104::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70880::DEBUG::2014-09-25 17:41:07,105::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70880::DEBUG::2014-09-25 17:41:07,105::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70880::DEBUG::2014-09-25 17:41:07,105::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70880::DEBUG::2014-09-25 17:41:07,105::task::993::Storage.TaskManager.Task::(_decref) Task=`09f4be80-ce86-4f1e-a1b5-aa5555380c81`::ref 0 aborting False >Thread-70880::DEBUG::2014-09-25 17:41:07,106::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70880::DEBUG::2014-09-25 17:41:07,106::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:07,187::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:07,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0140043 s, 73.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:08,259::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:08,259::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:08,260::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:08,260::__init__::232::IOProcess::(_processLogs) (109045) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:08,263::__init__::232::IOProcess::(_processLogs) (109045) Queuing response >Thread-79::DEBUG::2014-09-25 17:41:08,264::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:41:08,280::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00288618 s, 195 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:08,281::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:08,281::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:08,281::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:08,282::__init__::232::IOProcess::(_processLogs) (109046) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:08,286::__init__::232::IOProcess::(_processLogs) (109046) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:08,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:08,288::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:08,289::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:08,289::__init__::232::IOProcess::(_processLogs) (109047) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:08,289::__init__::232::IOProcess::(_processLogs) (109047) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:08,290::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:08,291::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:08,291::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:08,291::__init__::232::IOProcess::(_processLogs) (109048) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:08,291::__init__::232::IOProcess::(_processLogs) (109048) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:08,292::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:08,293::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:08,293::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:08,294::__init__::232::IOProcess::(_processLogs) (109049) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:08,294::__init__::232::IOProcess::(_processLogs) (109049) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:08,690::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:08,692::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70881::DEBUG::2014-09-25 17:41:08,693::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:09,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:09,252::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013537 s, 75.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:41:11,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:11,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133415 s, 76.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:11,713::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:11,714::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70882::DEBUG::2014-09-25 17:41:11,716::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:41:12,768::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:12,769::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:12,769::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:12,769::__init__::232::IOProcess::(_processLogs) (109050) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:12,774::__init__::232::IOProcess::(_processLogs) (109050) Queuing response >Thread-81::DEBUG::2014-09-25 17:41:12,775::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:41:12,792::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00674117 s, 55.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:12,793::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:12,794::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:12,794::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:12,794::__init__::232::IOProcess::(_processLogs) (109051) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:12,802::__init__::232::IOProcess::(_processLogs) (109051) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:13,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:13,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135253 s, 75.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:14,733::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:14,735::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70883::DEBUG::2014-09-25 17:41:14,737::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:15,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:15,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181864 s, 56.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:15,582::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:15,582::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:15,583::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:41:15,584::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:41:15,584::__init__::232::IOProcess::(_processLogs) (109052) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:15,593::__init__::232::IOProcess::(_processLogs) (109052) Queuing response >Thread-80::DEBUG::2014-09-25 17:41:15,594::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000534179 s, 708 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:15,595::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:15,595::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:15,595::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:15,596::__init__::232::IOProcess::(_processLogs) (109053) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:15,596::__init__::232::IOProcess::(_processLogs) (109053) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:17,136::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70884::DEBUG::2014-09-25 17:41:17,137::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:41:17,137::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70884::DEBUG::2014-09-25 17:41:17,139::task::595::Storage.TaskManager.Task::(_updateState) Task=`58210519-e750-4a21-9070-cae6288ae24a`::moving from state init -> state preparing >Thread-70884::INFO::2014-09-25 17:41:17,139::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70884::INFO::2014-09-25 17:41:17,139::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70884::DEBUG::2014-09-25 17:41:17,139::task::1191::Storage.TaskManager.Task::(prepare) Task=`58210519-e750-4a21-9070-cae6288ae24a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70884::DEBUG::2014-09-25 17:41:17,139::task::595::Storage.TaskManager.Task::(_updateState) Task=`58210519-e750-4a21-9070-cae6288ae24a`::moving from state preparing -> state finished >Thread-70884::DEBUG::2014-09-25 17:41:17,140::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70884::DEBUG::2014-09-25 17:41:17,140::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70884::DEBUG::2014-09-25 17:41:17,140::task::993::Storage.TaskManager.Task::(_decref) Task=`58210519-e750-4a21-9070-cae6288ae24a`::ref 0 aborting False >Thread-70884::DEBUG::2014-09-25 17:41:17,140::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70884::DEBUG::2014-09-25 17:41:17,140::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:17,155::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:17,156::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70885::DEBUG::2014-09-25 17:41:17,156::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70885::DEBUG::2014-09-25 17:41:17,157::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e9d8f81-90a3-4cc1-9efb-e8e3fb61981b`::moving from state init -> state preparing >Thread-70885::INFO::2014-09-25 17:41:17,157::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70885::DEBUG::2014-09-25 17:41:17,158::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`662dbcca-7139-4c93-9e82-46d3975af6a3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70885::DEBUG::2014-09-25 17:41:17,158::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70885::DEBUG::2014-09-25 17:41:17,158::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70885::DEBUG::2014-09-25 17:41:17,158::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`662dbcca-7139-4c93-9e82-46d3975af6a3`::Granted request >Thread-70885::DEBUG::2014-09-25 17:41:17,159::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3e9d8f81-90a3-4cc1-9efb-e8e3fb61981b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70885::DEBUG::2014-09-25 17:41:17,159::task::993::Storage.TaskManager.Task::(_decref) Task=`3e9d8f81-90a3-4cc1-9efb-e8e3fb61981b`::ref 1 aborting False >Thread-70885::INFO::2014-09-25 17:41:17,162::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70885::DEBUG::2014-09-25 17:41:17,162::task::1191::Storage.TaskManager.Task::(prepare) Task=`3e9d8f81-90a3-4cc1-9efb-e8e3fb61981b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70885::DEBUG::2014-09-25 17:41:17,162::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e9d8f81-90a3-4cc1-9efb-e8e3fb61981b`::moving from state preparing -> state finished >Thread-70885::DEBUG::2014-09-25 17:41:17,162::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70885::DEBUG::2014-09-25 17:41:17,162::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70885::DEBUG::2014-09-25 17:41:17,163::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70885::DEBUG::2014-09-25 17:41:17,163::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70885::DEBUG::2014-09-25 17:41:17,163::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70885::DEBUG::2014-09-25 17:41:17,163::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70885::DEBUG::2014-09-25 17:41:17,163::task::993::Storage.TaskManager.Task::(_decref) Task=`3e9d8f81-90a3-4cc1-9efb-e8e3fb61981b`::ref 0 aborting False >Thread-70885::DEBUG::2014-09-25 17:41:17,163::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70885::DEBUG::2014-09-25 17:41:17,164::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:17,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:17,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203854 s, 50.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:17,753::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:17,755::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70886::DEBUG::2014-09-25 17:41:17,757::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:41:18,294::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:18,294::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:18,295::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:18,295::__init__::232::IOProcess::(_processLogs) (109054) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:18,305::__init__::232::IOProcess::(_processLogs) (109054) Queuing response >Thread-79::DEBUG::2014-09-25 17:41:18,307::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:41:18,328::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00955424 s, 58.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:18,329::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:18,330::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:18,330::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:18,330::__init__::232::IOProcess::(_processLogs) (109055) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:18,347::__init__::232::IOProcess::(_processLogs) (109055) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:18,349::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:18,349::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:18,349::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:18,349::__init__::232::IOProcess::(_processLogs) (109056) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:18,350::__init__::232::IOProcess::(_processLogs) (109056) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:18,351::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:18,351::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:18,351::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:18,352::__init__::232::IOProcess::(_processLogs) (109057) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:18,352::__init__::232::IOProcess::(_processLogs) (109057) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:18,353::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:18,353::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:18,353::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:18,353::__init__::232::IOProcess::(_processLogs) (109058) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:18,354::__init__::232::IOProcess::(_processLogs) (109058) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:19,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:19,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145656 s, 70.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:20,772::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:20,773::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70887::DEBUG::2014-09-25 17:41:20,776::task::595::Storage.TaskManager.Task::(_updateState) Task=`46c929ce-bee1-46bd-954f-03d9a01d604e`::moving from state init -> state preparing >Thread-70887::INFO::2014-09-25 17:41:20,776::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70887::INFO::2014-09-25 17:41:20,777::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00955424', 'lastCheck': '2.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000534179', 'lastCheck': '5.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00674117', 'lastCheck': '8.0', 'valid': True}} >Thread-70887::DEBUG::2014-09-25 17:41:20,777::task::1191::Storage.TaskManager.Task::(prepare) Task=`46c929ce-bee1-46bd-954f-03d9a01d604e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00955424', 'lastCheck': '2.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000534179', 'lastCheck': '5.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00674117', 'lastCheck': '8.0', 'valid': True}} >Thread-70887::DEBUG::2014-09-25 17:41:20,777::task::595::Storage.TaskManager.Task::(_updateState) Task=`46c929ce-bee1-46bd-954f-03d9a01d604e`::moving from state preparing -> state finished >Thread-70887::DEBUG::2014-09-25 17:41:20,777::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70887::DEBUG::2014-09-25 17:41:20,777::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70887::DEBUG::2014-09-25 17:41:20,778::task::993::Storage.TaskManager.Task::(_decref) Task=`46c929ce-bee1-46bd-954f-03d9a01d604e`::ref 0 aborting False >Thread-70887::DEBUG::2014-09-25 17:41:20,780::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:20,808::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:20,809::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70888::DEBUG::2014-09-25 17:41:20,811::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:21,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:21,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205698 s, 49.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:22,805::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:22,805::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:22,806::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:22,806::__init__::232::IOProcess::(_processLogs) (109059) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:22,809::__init__::232::IOProcess::(_processLogs) (109059) Queuing response >Thread-81::DEBUG::2014-09-25 17:41:22,809::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:41:22,828::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00241575 s, 154 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:22,829::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:22,830::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:22,830::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:22,830::__init__::232::IOProcess::(_processLogs) (109060) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:22,836::__init__::232::IOProcess::(_processLogs) (109060) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:23,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:23,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136296 s, 75.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:23,872::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:23,873::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70889::DEBUG::2014-09-25 17:41:23,875::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:25,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:41:25,599::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:25,599::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:25,599::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:25,599::__init__::232::IOProcess::(_processLogs) (109061) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:25,600::__init__::232::IOProcess::(_processLogs) (109061) Queuing response >Thread-80::DEBUG::2014-09-25 17:41:25,601::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:41:25,610::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000532842 s, 709 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:25,611::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:25,611::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:25,612::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:25,612::__init__::232::IOProcess::(_processLogs) (109062) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:25,613::__init__::232::IOProcess::(_processLogs) (109062) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:25,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185435 s, 55.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:26,891::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:26,892::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70890::DEBUG::2014-09-25 17:41:26,894::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:27,192::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:27,194::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70891::DEBUG::2014-09-25 17:41:27,194::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70891::DEBUG::2014-09-25 17:41:27,196::task::595::Storage.TaskManager.Task::(_updateState) Task=`85a5d7b0-cdb4-4215-8dad-50cd08713499`::moving from state init -> state preparing >Thread-70891::INFO::2014-09-25 17:41:27,196::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70891::INFO::2014-09-25 17:41:27,197::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70891::DEBUG::2014-09-25 17:41:27,197::task::1191::Storage.TaskManager.Task::(prepare) Task=`85a5d7b0-cdb4-4215-8dad-50cd08713499`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70891::DEBUG::2014-09-25 17:41:27,197::task::595::Storage.TaskManager.Task::(_updateState) Task=`85a5d7b0-cdb4-4215-8dad-50cd08713499`::moving from state preparing -> state finished >Thread-70891::DEBUG::2014-09-25 17:41:27,198::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70891::DEBUG::2014-09-25 17:41:27,198::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70891::DEBUG::2014-09-25 17:41:27,198::task::993::Storage.TaskManager.Task::(_decref) Task=`85a5d7b0-cdb4-4215-8dad-50cd08713499`::ref 0 aborting False >Thread-70891::DEBUG::2014-09-25 17:41:27,198::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70891::DEBUG::2014-09-25 17:41:27,199::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:27,212::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70892::DEBUG::2014-09-25 17:41:27,213::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:41:27,213::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70892::DEBUG::2014-09-25 17:41:27,215::task::595::Storage.TaskManager.Task::(_updateState) Task=`69e97232-ad92-4fda-83bd-1156983ed760`::moving from state init -> state preparing >Thread-70892::INFO::2014-09-25 17:41:27,216::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70892::DEBUG::2014-09-25 17:41:27,216::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1e22e6c4-92f5-4b3f-8430-37edfd48a59d`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70892::DEBUG::2014-09-25 17:41:27,217::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70892::DEBUG::2014-09-25 17:41:27,217::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70892::DEBUG::2014-09-25 17:41:27,217::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1e22e6c4-92f5-4b3f-8430-37edfd48a59d`::Granted request >Thread-70892::DEBUG::2014-09-25 17:41:27,218::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`69e97232-ad92-4fda-83bd-1156983ed760`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70892::DEBUG::2014-09-25 17:41:27,218::task::993::Storage.TaskManager.Task::(_decref) Task=`69e97232-ad92-4fda-83bd-1156983ed760`::ref 1 aborting False >Thread-70892::INFO::2014-09-25 17:41:27,222::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70892::DEBUG::2014-09-25 17:41:27,223::task::1191::Storage.TaskManager.Task::(prepare) Task=`69e97232-ad92-4fda-83bd-1156983ed760`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70892::DEBUG::2014-09-25 17:41:27,223::task::595::Storage.TaskManager.Task::(_updateState) Task=`69e97232-ad92-4fda-83bd-1156983ed760`::moving from state preparing -> state finished >Thread-70892::DEBUG::2014-09-25 17:41:27,223::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70892::DEBUG::2014-09-25 17:41:27,223::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70892::DEBUG::2014-09-25 17:41:27,224::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70892::DEBUG::2014-09-25 17:41:27,224::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70892::DEBUG::2014-09-25 17:41:27,224::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70892::DEBUG::2014-09-25 17:41:27,224::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70892::DEBUG::2014-09-25 17:41:27,225::task::993::Storage.TaskManager.Task::(_decref) Task=`69e97232-ad92-4fda-83bd-1156983ed760`::ref 0 aborting False >Thread-70892::DEBUG::2014-09-25 17:41:27,225::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70892::DEBUG::2014-09-25 17:41:27,226::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:27,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:27,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150354 s, 68.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:28,356::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:28,356::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:28,357::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:28,357::__init__::232::IOProcess::(_processLogs) (109063) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:28,359::__init__::232::IOProcess::(_processLogs) (109063) Queuing response >Thread-79::DEBUG::2014-09-25 17:41:28,360::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:41:28,375::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00168921 s, 333 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:28,376::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:28,376::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:28,377::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:28,378::__init__::232::IOProcess::(_processLogs) (109064) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:28,378::__init__::232::IOProcess::(_processLogs) (109064) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:28,379::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:28,379::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:28,380::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:28,380::__init__::232::IOProcess::(_processLogs) (109065) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:28,380::__init__::232::IOProcess::(_processLogs) (109065) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:28,381::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:28,381::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:28,382::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:28,382::__init__::232::IOProcess::(_processLogs) (109066) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:28,383::__init__::232::IOProcess::(_processLogs) (109066) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:28,383::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:28,383::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:28,384::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:28,385::__init__::232::IOProcess::(_processLogs) (109067) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:28,385::__init__::232::IOProcess::(_processLogs) (109067) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:29,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:29,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215973 s, 47.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:29,910::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:29,911::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70893::DEBUG::2014-09-25 17:41:29,913::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:31,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:31,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162721 s, 62.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:32,838::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:32,839::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:32,839::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:32,839::__init__::232::IOProcess::(_processLogs) (109068) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:32,842::__init__::232::IOProcess::(_processLogs) (109068) Queuing response >Thread-81::DEBUG::2014-09-25 17:41:32,843::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:41:32,861::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00759159 s, 48.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:32,862::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:32,864::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:32,864::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:32,865::__init__::232::IOProcess::(_processLogs) (109069) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:32,868::__init__::232::IOProcess::(_processLogs) (109069) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:32,929::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:32,931::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70894::DEBUG::2014-09-25 17:41:32,934::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:33,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:33,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133604 s, 76.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:35,615::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:35,615::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:35,616::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:35,616::__init__::232::IOProcess::(_processLogs) (109070) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:35,616::__init__::232::IOProcess::(_processLogs) (109070) Queuing response >Thread-80::DEBUG::2014-09-25 17:41:35,617::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:41:35,627::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000565086 s, 669 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:35,628::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:35,629::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:35,629::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:35,629::__init__::232::IOProcess::(_processLogs) (109071) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:35,630::__init__::232::IOProcess::(_processLogs) (109071) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:35,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:35,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191124 s, 53.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:35,948::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:35,949::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70895::DEBUG::2014-09-25 17:41:35,952::task::595::Storage.TaskManager.Task::(_updateState) Task=`04e35435-5786-4fc7-83d7-d9cfc7583db9`::moving from state init -> state preparing >Thread-70895::INFO::2014-09-25 17:41:35,952::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70895::INFO::2014-09-25 17:41:35,953::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00168921', 'lastCheck': '7.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000565086', 'lastCheck': '0.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00759159', 'lastCheck': '3.1', 'valid': True}} >Thread-70895::DEBUG::2014-09-25 17:41:35,953::task::1191::Storage.TaskManager.Task::(prepare) Task=`04e35435-5786-4fc7-83d7-d9cfc7583db9`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00168921', 'lastCheck': '7.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000565086', 'lastCheck': '0.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00759159', 'lastCheck': '3.1', 'valid': True}} >Thread-70895::DEBUG::2014-09-25 17:41:35,953::task::595::Storage.TaskManager.Task::(_updateState) Task=`04e35435-5786-4fc7-83d7-d9cfc7583db9`::moving from state preparing -> state finished >Thread-70895::DEBUG::2014-09-25 17:41:35,953::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70895::DEBUG::2014-09-25 17:41:35,954::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70895::DEBUG::2014-09-25 17:41:35,954::task::993::Storage.TaskManager.Task::(_decref) Task=`04e35435-5786-4fc7-83d7-d9cfc7583db9`::ref 0 aborting False >Thread-70895::DEBUG::2014-09-25 17:41:35,956::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:35,976::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:35,977::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70896::DEBUG::2014-09-25 17:41:35,979::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:37,254::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70897::DEBUG::2014-09-25 17:41:37,255::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:41:37,255::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70897::DEBUG::2014-09-25 17:41:37,257::task::595::Storage.TaskManager.Task::(_updateState) Task=`a053b153-4b1a-4b79-a9f5-4bbd7c29e388`::moving from state init -> state preparing >Thread-70897::INFO::2014-09-25 17:41:37,258::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70897::INFO::2014-09-25 17:41:37,258::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70897::DEBUG::2014-09-25 17:41:37,259::task::1191::Storage.TaskManager.Task::(prepare) Task=`a053b153-4b1a-4b79-a9f5-4bbd7c29e388`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70897::DEBUG::2014-09-25 17:41:37,259::task::595::Storage.TaskManager.Task::(_updateState) Task=`a053b153-4b1a-4b79-a9f5-4bbd7c29e388`::moving from state preparing -> state finished >Thread-70897::DEBUG::2014-09-25 17:41:37,259::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70897::DEBUG::2014-09-25 17:41:37,259::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70897::DEBUG::2014-09-25 17:41:37,259::task::993::Storage.TaskManager.Task::(_decref) Task=`a053b153-4b1a-4b79-a9f5-4bbd7c29e388`::ref 0 aborting False >Thread-70897::DEBUG::2014-09-25 17:41:37,260::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70897::DEBUG::2014-09-25 17:41:37,260::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:37,274::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:37,275::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70898::DEBUG::2014-09-25 17:41:37,275::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70898::DEBUG::2014-09-25 17:41:37,277::task::595::Storage.TaskManager.Task::(_updateState) Task=`e38c28bf-9d6e-40dd-85ad-a68dddecbfbe`::moving from state init -> state preparing >Thread-70898::INFO::2014-09-25 17:41:37,277::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70898::DEBUG::2014-09-25 17:41:37,278::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ddbbc393-7674-4796-b1b0-234726404e67`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70898::DEBUG::2014-09-25 17:41:37,278::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70898::DEBUG::2014-09-25 17:41:37,278::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70898::DEBUG::2014-09-25 17:41:37,278::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ddbbc393-7674-4796-b1b0-234726404e67`::Granted request >Thread-70898::DEBUG::2014-09-25 17:41:37,279::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e38c28bf-9d6e-40dd-85ad-a68dddecbfbe`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70898::DEBUG::2014-09-25 17:41:37,279::task::993::Storage.TaskManager.Task::(_decref) Task=`e38c28bf-9d6e-40dd-85ad-a68dddecbfbe`::ref 1 aborting False >Thread-70898::INFO::2014-09-25 17:41:37,283::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70898::DEBUG::2014-09-25 17:41:37,284::task::1191::Storage.TaskManager.Task::(prepare) Task=`e38c28bf-9d6e-40dd-85ad-a68dddecbfbe`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70898::DEBUG::2014-09-25 17:41:37,284::task::595::Storage.TaskManager.Task::(_updateState) Task=`e38c28bf-9d6e-40dd-85ad-a68dddecbfbe`::moving from state preparing -> state finished >Thread-70898::DEBUG::2014-09-25 17:41:37,284::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70898::DEBUG::2014-09-25 17:41:37,284::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70898::DEBUG::2014-09-25 17:41:37,285::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70898::DEBUG::2014-09-25 17:41:37,285::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70898::DEBUG::2014-09-25 17:41:37,285::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70898::DEBUG::2014-09-25 17:41:37,286::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70898::DEBUG::2014-09-25 17:41:37,286::task::993::Storage.TaskManager.Task::(_decref) Task=`e38c28bf-9d6e-40dd-85ad-a68dddecbfbe`::ref 0 aborting False >Thread-70898::DEBUG::2014-09-25 17:41:37,286::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70898::DEBUG::2014-09-25 17:41:37,287::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:37,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:38,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.263918 s, 3.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:38,386::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:38,386::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:38,386::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:38,386::__init__::232::IOProcess::(_processLogs) (109072) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:38,389::__init__::232::IOProcess::(_processLogs) (109072) Queuing response >Thread-79::DEBUG::2014-09-25 17:41:38,390::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:41:38,405::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00385086 s, 146 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:38,406::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:38,407::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:38,407::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:38,407::__init__::232::IOProcess::(_processLogs) (109073) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:38,411::__init__::232::IOProcess::(_processLogs) (109073) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:38,412::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:38,413::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:38,413::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:38,413::__init__::232::IOProcess::(_processLogs) (109074) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:38,413::__init__::232::IOProcess::(_processLogs) (109074) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:38,414::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:38,415::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:38,415::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:38,416::__init__::232::IOProcess::(_processLogs) (109075) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:38,416::__init__::232::IOProcess::(_processLogs) (109075) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:38,416::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:38,416::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:38,417::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:38,417::__init__::232::IOProcess::(_processLogs) (109076) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:38,417::__init__::232::IOProcess::(_processLogs) (109076) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:39,041::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:39,042::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70899::DEBUG::2014-09-25 17:41:39,044::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:40,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:40,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148537 s, 68.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:42,083::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:42,084::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70900::DEBUG::2014-09-25 17:41:42,086::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:42,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:42,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174905 s, 58.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:42,870::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:42,870::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:42,870::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:42,871::__init__::232::IOProcess::(_processLogs) (109077) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:42,873::__init__::232::IOProcess::(_processLogs) (109077) Queuing response >Thread-81::DEBUG::2014-09-25 17:41:42,874::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:41:42,892::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00394476 s, 94.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:42,893::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:42,893::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:42,893::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:42,894::__init__::232::IOProcess::(_processLogs) (109078) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:42,897::__init__::232::IOProcess::(_processLogs) (109078) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:44,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:44,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0298077 s, 34.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:45,100::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:45,102::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70901::DEBUG::2014-09-25 17:41:45,104::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:41:45,632::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:45,632::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:45,633::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:45,633::__init__::232::IOProcess::(_processLogs) (109079) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:45,633::__init__::232::IOProcess::(_processLogs) (109079) Queuing response >Thread-80::DEBUG::2014-09-25 17:41:45,634::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:41:45,643::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000370382 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:45,644::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:45,645::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:45,645::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:45,645::__init__::232::IOProcess::(_processLogs) (109080) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:45,646::__init__::232::IOProcess::(_processLogs) (109080) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:46,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:46,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0253967 s, 40.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:47,314::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:47,315::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70902::DEBUG::2014-09-25 17:41:47,315::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70902::DEBUG::2014-09-25 17:41:47,317::task::595::Storage.TaskManager.Task::(_updateState) Task=`88e5031e-68ae-487a-b8ed-ca3eae862f6b`::moving from state init -> state preparing >Thread-70902::INFO::2014-09-25 17:41:47,317::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70902::INFO::2014-09-25 17:41:47,317::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70902::DEBUG::2014-09-25 17:41:47,317::task::1191::Storage.TaskManager.Task::(prepare) Task=`88e5031e-68ae-487a-b8ed-ca3eae862f6b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70902::DEBUG::2014-09-25 17:41:47,318::task::595::Storage.TaskManager.Task::(_updateState) Task=`88e5031e-68ae-487a-b8ed-ca3eae862f6b`::moving from state preparing -> state finished >Thread-70902::DEBUG::2014-09-25 17:41:47,318::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70902::DEBUG::2014-09-25 17:41:47,318::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70902::DEBUG::2014-09-25 17:41:47,318::task::993::Storage.TaskManager.Task::(_decref) Task=`88e5031e-68ae-487a-b8ed-ca3eae862f6b`::ref 0 aborting False >Thread-70902::DEBUG::2014-09-25 17:41:47,318::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70902::DEBUG::2014-09-25 17:41:47,318::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:47,331::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70903::DEBUG::2014-09-25 17:41:47,332::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:41:47,332::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70903::DEBUG::2014-09-25 17:41:47,333::task::595::Storage.TaskManager.Task::(_updateState) Task=`162690d1-ba51-46f4-a6da-99771cb4eff6`::moving from state init -> state preparing >Thread-70903::INFO::2014-09-25 17:41:47,334::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70903::DEBUG::2014-09-25 17:41:47,334::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b03e442d-6554-4866-837d-81cabb50e668`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70903::DEBUG::2014-09-25 17:41:47,334::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70903::DEBUG::2014-09-25 17:41:47,335::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70903::DEBUG::2014-09-25 17:41:47,335::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b03e442d-6554-4866-837d-81cabb50e668`::Granted request >Thread-70903::DEBUG::2014-09-25 17:41:47,335::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`162690d1-ba51-46f4-a6da-99771cb4eff6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70903::DEBUG::2014-09-25 17:41:47,335::task::993::Storage.TaskManager.Task::(_decref) Task=`162690d1-ba51-46f4-a6da-99771cb4eff6`::ref 1 aborting False >Thread-70903::INFO::2014-09-25 17:41:47,338::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70903::DEBUG::2014-09-25 17:41:47,338::task::1191::Storage.TaskManager.Task::(prepare) Task=`162690d1-ba51-46f4-a6da-99771cb4eff6`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70903::DEBUG::2014-09-25 17:41:47,338::task::595::Storage.TaskManager.Task::(_updateState) Task=`162690d1-ba51-46f4-a6da-99771cb4eff6`::moving from state preparing -> state finished >Thread-70903::DEBUG::2014-09-25 17:41:47,338::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70903::DEBUG::2014-09-25 17:41:47,338::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70903::DEBUG::2014-09-25 17:41:47,339::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70903::DEBUG::2014-09-25 17:41:47,339::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70903::DEBUG::2014-09-25 17:41:47,339::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70903::DEBUG::2014-09-25 17:41:47,339::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70903::DEBUG::2014-09-25 17:41:47,339::task::993::Storage.TaskManager.Task::(_decref) Task=`162690d1-ba51-46f4-a6da-99771cb4eff6`::ref 0 aborting False >Thread-70903::DEBUG::2014-09-25 17:41:47,340::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70903::DEBUG::2014-09-25 17:41:47,340::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:48,119::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:48,121::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70904::DEBUG::2014-09-25 17:41:48,123::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:41:48,419::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:48,419::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:48,419::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:48,421::__init__::232::IOProcess::(_processLogs) (109081) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 17:41:48,421::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:41:48,422::__init__::232::IOProcess::(_processLogs) (109081) Queuing response >Thread-79::DEBUG::2014-09-25 17:41:48,431::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000417297 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:48,432::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:48,432::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:48,432::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:48,432::__init__::232::IOProcess::(_processLogs) (109082) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:48,433::__init__::232::IOProcess::(_processLogs) (109082) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:48,434::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:48,435::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:48,435::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:48,436::__init__::232::IOProcess::(_processLogs) (109083) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:48,436::__init__::232::IOProcess::(_processLogs) (109083) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:48,436::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:48,437::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:48,437::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:48,437::__init__::232::IOProcess::(_processLogs) (109084) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:48,438::__init__::232::IOProcess::(_processLogs) (109084) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:48,439::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:48,439::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:48,439::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:48,439::__init__::232::IOProcess::(_processLogs) (109085) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:48,440::__init__::232::IOProcess::(_processLogs) (109085) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:48,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:48,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183734 s, 55.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:41:50,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:50,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162551 s, 63.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:51,138::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:51,140::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70905::DEBUG::2014-09-25 17:41:51,142::task::595::Storage.TaskManager.Task::(_updateState) Task=`20a90a9a-5d7e-44a9-b44a-b57cdb96161f`::moving from state init -> state preparing >Thread-70905::INFO::2014-09-25 17:41:51,143::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70905::INFO::2014-09-25 17:41:51,143::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000417297', 'lastCheck': '2.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000370382', 'lastCheck': '5.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00394476', 'lastCheck': '8.2', 'valid': True}} >Thread-70905::DEBUG::2014-09-25 17:41:51,143::task::1191::Storage.TaskManager.Task::(prepare) Task=`20a90a9a-5d7e-44a9-b44a-b57cdb96161f`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000417297', 'lastCheck': '2.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000370382', 'lastCheck': '5.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00394476', 'lastCheck': '8.2', 'valid': True}} >Thread-70905::DEBUG::2014-09-25 17:41:51,144::task::595::Storage.TaskManager.Task::(_updateState) Task=`20a90a9a-5d7e-44a9-b44a-b57cdb96161f`::moving from state preparing -> state finished >Thread-70905::DEBUG::2014-09-25 17:41:51,144::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70905::DEBUG::2014-09-25 17:41:51,144::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70905::DEBUG::2014-09-25 17:41:51,144::task::993::Storage.TaskManager.Task::(_decref) Task=`20a90a9a-5d7e-44a9-b44a-b57cdb96161f`::ref 0 aborting False >Thread-70905::DEBUG::2014-09-25 17:41:51,146::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:51,168::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:51,169::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70906::DEBUG::2014-09-25 17:41:51,171::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:52,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:52,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185746 s, 55.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:52,900::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:52,900::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:52,900::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:52,900::__init__::232::IOProcess::(_processLogs) (109086) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:52,917::__init__::232::IOProcess::(_processLogs) (109086) Queuing response >Thread-81::DEBUG::2014-09-25 17:41:52,918::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:41:52,939::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00899796 s, 41.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:52,940::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:52,941::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:52,941::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:52,941::__init__::232::IOProcess::(_processLogs) (109087) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:52,946::__init__::232::IOProcess::(_processLogs) (109087) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:54,225::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:54,227::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70907::DEBUG::2014-09-25 17:41:54,229::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:41:54,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:54,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144077 s, 71.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:55,649::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:55,649::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:55,649::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:55,649::__init__::232::IOProcess::(_processLogs) (109088) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:55,650::__init__::232::IOProcess::(_processLogs) (109088) Queuing response >Thread-80::DEBUG::2014-09-25 17:41:55,651::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:41:55,660::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000432424 s, 874 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:55,661::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:55,661::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:55,662::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:55,662::__init__::232::IOProcess::(_processLogs) (109089) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:55,662::__init__::232::IOProcess::(_processLogs) (109089) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:56,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:56,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165414 s, 61.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:57,246::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:57,247::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70908::DEBUG::2014-09-25 17:41:57,249::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:57,368::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:57,370::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70909::DEBUG::2014-09-25 17:41:57,370::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70909::DEBUG::2014-09-25 17:41:57,371::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d76bafa-aa5b-4a50-a787-dc1d95f99409`::moving from state init -> state preparing >Thread-70909::INFO::2014-09-25 17:41:57,372::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70909::INFO::2014-09-25 17:41:57,372::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70909::DEBUG::2014-09-25 17:41:57,373::task::1191::Storage.TaskManager.Task::(prepare) Task=`8d76bafa-aa5b-4a50-a787-dc1d95f99409`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70909::DEBUG::2014-09-25 17:41:57,373::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d76bafa-aa5b-4a50-a787-dc1d95f99409`::moving from state preparing -> state finished >Thread-70909::DEBUG::2014-09-25 17:41:57,373::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70909::DEBUG::2014-09-25 17:41:57,373::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70909::DEBUG::2014-09-25 17:41:57,373::task::993::Storage.TaskManager.Task::(_decref) Task=`8d76bafa-aa5b-4a50-a787-dc1d95f99409`::ref 0 aborting False >Thread-70909::DEBUG::2014-09-25 17:41:57,374::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70909::DEBUG::2014-09-25 17:41:57,374::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:41:57,387::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:41:57,389::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70910::DEBUG::2014-09-25 17:41:57,389::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70910::DEBUG::2014-09-25 17:41:57,390::task::595::Storage.TaskManager.Task::(_updateState) Task=`b19f4435-f8b1-4cd7-b8fd-3c5c338cc14e`::moving from state init -> state preparing >Thread-70910::INFO::2014-09-25 17:41:57,391::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70910::DEBUG::2014-09-25 17:41:57,391::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c97222e6-241d-4169-9962-6db4ac6b38fe`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70910::DEBUG::2014-09-25 17:41:57,392::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70910::DEBUG::2014-09-25 17:41:57,392::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70910::DEBUG::2014-09-25 17:41:57,392::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c97222e6-241d-4169-9962-6db4ac6b38fe`::Granted request >Thread-70910::DEBUG::2014-09-25 17:41:57,393::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b19f4435-f8b1-4cd7-b8fd-3c5c338cc14e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70910::DEBUG::2014-09-25 17:41:57,393::task::993::Storage.TaskManager.Task::(_decref) Task=`b19f4435-f8b1-4cd7-b8fd-3c5c338cc14e`::ref 1 aborting False >Thread-70910::INFO::2014-09-25 17:41:57,397::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70910::DEBUG::2014-09-25 17:41:57,398::task::1191::Storage.TaskManager.Task::(prepare) Task=`b19f4435-f8b1-4cd7-b8fd-3c5c338cc14e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70910::DEBUG::2014-09-25 17:41:57,398::task::595::Storage.TaskManager.Task::(_updateState) Task=`b19f4435-f8b1-4cd7-b8fd-3c5c338cc14e`::moving from state preparing -> state finished >Thread-70910::DEBUG::2014-09-25 17:41:57,398::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70910::DEBUG::2014-09-25 17:41:57,398::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70910::DEBUG::2014-09-25 17:41:57,399::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70910::DEBUG::2014-09-25 17:41:57,399::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70910::DEBUG::2014-09-25 17:41:57,399::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70910::DEBUG::2014-09-25 17:41:57,400::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70910::DEBUG::2014-09-25 17:41:57,400::task::993::Storage.TaskManager.Task::(_decref) Task=`b19f4435-f8b1-4cd7-b8fd-3c5c338cc14e`::ref 0 aborting False >Thread-70910::DEBUG::2014-09-25 17:41:57,400::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70910::DEBUG::2014-09-25 17:41:57,401::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:41:58,442::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:58,442::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:58,442::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:58,443::__init__::232::IOProcess::(_processLogs) (109090) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:58,447::__init__::232::IOProcess::(_processLogs) (109090) Queuing response >Thread-79::DEBUG::2014-09-25 17:41:58,448::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:41:58,459::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00209693 s, 268 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:41:58,460::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:58,460::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:58,461::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:58,461::__init__::232::IOProcess::(_processLogs) (109091) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:41:58,465::__init__::232::IOProcess::(_processLogs) (109091) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:58,466::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:58,466::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:58,467::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:58,467::__init__::232::IOProcess::(_processLogs) (109092) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:58,468::__init__::232::IOProcess::(_processLogs) (109092) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:58,468::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:58,468::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:58,469::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:58,470::__init__::232::IOProcess::(_processLogs) (109093) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:58,470::__init__::232::IOProcess::(_processLogs) (109093) Queuing response >Thread-29::DEBUG::2014-09-25 17:41:58,470::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:41:58,471::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:41:58,471::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:41:58,472::__init__::232::IOProcess::(_processLogs) (109094) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:41:58,472::__init__::232::IOProcess::(_processLogs) (109094) Queuing response >Dummy-148::DEBUG::2014-09-25 17:41:58,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:41:58,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135023 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:00,266::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:00,267::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70911::DEBUG::2014-09-25 17:42:00,269::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:00,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:00,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218566 s, 46.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:42:02,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:42:02,948::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:02,948::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:02,948::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:02,948::__init__::232::IOProcess::(_processLogs) (109095) Got request for method 'statvfs' >Dummy-148::DEBUG::2014-09-25 17:42:02,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177745 s, 57.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:02,957::__init__::232::IOProcess::(_processLogs) (109095) Queuing response >Thread-81::DEBUG::2014-09-25 17:42:02,958::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:42:02,981::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00548505 s, 67.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:02,984::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:02,985::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:02,986::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:02,986::__init__::232::IOProcess::(_processLogs) (109096) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:02,987::__init__::232::IOProcess::(_processLogs) (109096) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:03,285::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:03,287::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70912::DEBUG::2014-09-25 17:42:03,289::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:04,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:05,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016691 s, 61.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:05,664::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:05,664::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:05,665::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:05,665::__init__::232::IOProcess::(_processLogs) (109097) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:05,665::__init__::232::IOProcess::(_processLogs) (109097) Queuing response >Thread-80::DEBUG::2014-09-25 17:42:05,667::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:42:05,675::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000369191 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:05,677::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:05,677::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:05,677::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:05,678::__init__::232::IOProcess::(_processLogs) (109098) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:05,678::__init__::232::IOProcess::(_processLogs) (109098) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:06,302::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:06,304::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70913::DEBUG::2014-09-25 17:42:06,306::task::595::Storage.TaskManager.Task::(_updateState) Task=`24ba51be-9666-441c-8ad1-aa90491a4756`::moving from state init -> state preparing >Thread-70913::INFO::2014-09-25 17:42:06,307::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70913::INFO::2014-09-25 17:42:06,307::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00209693', 'lastCheck': '7.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000369191', 'lastCheck': '0.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00548505', 'lastCheck': '3.3', 'valid': True}} >Thread-70913::DEBUG::2014-09-25 17:42:06,307::task::1191::Storage.TaskManager.Task::(prepare) Task=`24ba51be-9666-441c-8ad1-aa90491a4756`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00209693', 'lastCheck': '7.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000369191', 'lastCheck': '0.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00548505', 'lastCheck': '3.3', 'valid': True}} >Thread-70913::DEBUG::2014-09-25 17:42:06,308::task::595::Storage.TaskManager.Task::(_updateState) Task=`24ba51be-9666-441c-8ad1-aa90491a4756`::moving from state preparing -> state finished >Thread-70913::DEBUG::2014-09-25 17:42:06,308::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70913::DEBUG::2014-09-25 17:42:06,308::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70913::DEBUG::2014-09-25 17:42:06,308::task::993::Storage.TaskManager.Task::(_decref) Task=`24ba51be-9666-441c-8ad1-aa90491a4756`::ref 0 aborting False >Thread-70913::DEBUG::2014-09-25 17:42:06,311::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:06,331::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:06,332::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70914::DEBUG::2014-09-25 17:42:06,334::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:07,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:07,064::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135719 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:07,428::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70915::DEBUG::2014-09-25 17:42:07,429::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:42:07,429::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70915::DEBUG::2014-09-25 17:42:07,432::task::595::Storage.TaskManager.Task::(_updateState) Task=`9451769a-ecc5-4d2a-9355-b6b8f3dbcc28`::moving from state init -> state preparing >Thread-70915::INFO::2014-09-25 17:42:07,432::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70915::INFO::2014-09-25 17:42:07,433::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70915::DEBUG::2014-09-25 17:42:07,433::task::1191::Storage.TaskManager.Task::(prepare) Task=`9451769a-ecc5-4d2a-9355-b6b8f3dbcc28`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70915::DEBUG::2014-09-25 17:42:07,433::task::595::Storage.TaskManager.Task::(_updateState) Task=`9451769a-ecc5-4d2a-9355-b6b8f3dbcc28`::moving from state preparing -> state finished >Thread-70915::DEBUG::2014-09-25 17:42:07,433::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70915::DEBUG::2014-09-25 17:42:07,434::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70915::DEBUG::2014-09-25 17:42:07,434::task::993::Storage.TaskManager.Task::(_decref) Task=`9451769a-ecc5-4d2a-9355-b6b8f3dbcc28`::ref 0 aborting False >Thread-70915::DEBUG::2014-09-25 17:42:07,434::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70915::DEBUG::2014-09-25 17:42:07,435::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:07,448::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:07,449::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70916::DEBUG::2014-09-25 17:42:07,449::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70916::DEBUG::2014-09-25 17:42:07,451::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ab77353-28a0-46eb-a27c-8f15b1cb3918`::moving from state init -> state preparing >Thread-70916::INFO::2014-09-25 17:42:07,451::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70916::DEBUG::2014-09-25 17:42:07,452::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d5601c61-544f-4991-8880-15552d2e8b4a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70916::DEBUG::2014-09-25 17:42:07,452::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70916::DEBUG::2014-09-25 17:42:07,453::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70916::DEBUG::2014-09-25 17:42:07,453::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d5601c61-544f-4991-8880-15552d2e8b4a`::Granted request >Thread-70916::DEBUG::2014-09-25 17:42:07,454::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0ab77353-28a0-46eb-a27c-8f15b1cb3918`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70916::DEBUG::2014-09-25 17:42:07,454::task::993::Storage.TaskManager.Task::(_decref) Task=`0ab77353-28a0-46eb-a27c-8f15b1cb3918`::ref 1 aborting False >Thread-70916::INFO::2014-09-25 17:42:07,458::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70916::DEBUG::2014-09-25 17:42:07,458::task::1191::Storage.TaskManager.Task::(prepare) Task=`0ab77353-28a0-46eb-a27c-8f15b1cb3918`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70916::DEBUG::2014-09-25 17:42:07,459::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ab77353-28a0-46eb-a27c-8f15b1cb3918`::moving from state preparing -> state finished >Thread-70916::DEBUG::2014-09-25 17:42:07,459::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70916::DEBUG::2014-09-25 17:42:07,459::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70916::DEBUG::2014-09-25 17:42:07,460::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70916::DEBUG::2014-09-25 17:42:07,460::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70916::DEBUG::2014-09-25 17:42:07,460::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70916::DEBUG::2014-09-25 17:42:07,460::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70916::DEBUG::2014-09-25 17:42:07,461::task::993::Storage.TaskManager.Task::(_decref) Task=`0ab77353-28a0-46eb-a27c-8f15b1cb3918`::ref 0 aborting False >Thread-70916::DEBUG::2014-09-25 17:42:07,461::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70916::DEBUG::2014-09-25 17:42:07,462::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:42:08,473::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:08,473::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:08,474::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:08,474::__init__::232::IOProcess::(_processLogs) (109099) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:08,476::__init__::232::IOProcess::(_processLogs) (109099) Queuing response >Thread-79::DEBUG::2014-09-25 17:42:08,477::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:42:08,493::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00251027 s, 224 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:08,494::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:08,494::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:08,494::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:08,495::__init__::232::IOProcess::(_processLogs) (109100) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:08,495::__init__::232::IOProcess::(_processLogs) (109100) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:08,497::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:08,497::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:08,498::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:08,498::__init__::232::IOProcess::(_processLogs) (109101) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:08,499::__init__::232::IOProcess::(_processLogs) (109101) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:08,499::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:08,500::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:08,500::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:08,500::__init__::232::IOProcess::(_processLogs) (109102) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:08,501::__init__::232::IOProcess::(_processLogs) (109102) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:08,501::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:08,501::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:08,502::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:08,502::__init__::232::IOProcess::(_processLogs) (109103) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:08,502::__init__::232::IOProcess::(_processLogs) (109103) Queuing response >Dummy-148::DEBUG::2014-09-25 17:42:09,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:09,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015646 s, 65.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:09,391::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:09,393::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70917::DEBUG::2014-09-25 17:42:09,395::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:11,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:11,178::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0311688 s, 32.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:12,411::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:12,412::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70918::DEBUG::2014-09-25 17:42:12,414::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:42:12,989::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:12,990::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:12,990::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:12,990::__init__::232::IOProcess::(_processLogs) (109104) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:12,993::__init__::232::IOProcess::(_processLogs) (109104) Queuing response >Thread-81::DEBUG::2014-09-25 17:42:12,995::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:42:13,011::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00452863 s, 81.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:13,013::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:13,013::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:13,013::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:13,013::__init__::232::IOProcess::(_processLogs) (109105) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:13,020::__init__::232::IOProcess::(_processLogs) (109105) Queuing response >Dummy-148::DEBUG::2014-09-25 17:42:13,210::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:13,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192575 s, 53.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:42:15,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:15,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0206245 s, 49.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:15,432::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:15,433::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70919::DEBUG::2014-09-25 17:42:15,435::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:42:15,680::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:15,680::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:15,681::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:15,681::__init__::232::IOProcess::(_processLogs) (109106) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:15,682::__init__::232::IOProcess::(_processLogs) (109106) Queuing response >Thread-80::DEBUG::2014-09-25 17:42:15,683::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:42:15,692::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000531464 s, 711 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:15,693::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:15,694::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:15,694::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:15,694::__init__::232::IOProcess::(_processLogs) (109107) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:15,695::__init__::232::IOProcess::(_processLogs) (109107) Queuing response >Dummy-148::DEBUG::2014-09-25 17:42:17,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:17,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204977 s, 50.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:17,491::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70920::DEBUG::2014-09-25 17:42:17,492::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:42:17,492::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70920::DEBUG::2014-09-25 17:42:17,494::task::595::Storage.TaskManager.Task::(_updateState) Task=`af92a894-deae-4979-a0e8-90030026741e`::moving from state init -> state preparing >Thread-70920::INFO::2014-09-25 17:42:17,495::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70920::INFO::2014-09-25 17:42:17,495::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70920::DEBUG::2014-09-25 17:42:17,495::task::1191::Storage.TaskManager.Task::(prepare) Task=`af92a894-deae-4979-a0e8-90030026741e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70920::DEBUG::2014-09-25 17:42:17,496::task::595::Storage.TaskManager.Task::(_updateState) Task=`af92a894-deae-4979-a0e8-90030026741e`::moving from state preparing -> state finished >Thread-70920::DEBUG::2014-09-25 17:42:17,496::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70920::DEBUG::2014-09-25 17:42:17,496::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70920::DEBUG::2014-09-25 17:42:17,496::task::993::Storage.TaskManager.Task::(_decref) Task=`af92a894-deae-4979-a0e8-90030026741e`::ref 0 aborting False >Thread-70920::DEBUG::2014-09-25 17:42:17,497::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70920::DEBUG::2014-09-25 17:42:17,497::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:17,512::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:17,513::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70921::DEBUG::2014-09-25 17:42:17,513::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70921::DEBUG::2014-09-25 17:42:17,514::task::595::Storage.TaskManager.Task::(_updateState) Task=`a73b058d-37ce-4959-a573-3bd95b83c435`::moving from state init -> state preparing >Thread-70921::INFO::2014-09-25 17:42:17,515::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70921::DEBUG::2014-09-25 17:42:17,515::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aa743a85-d27d-4fbe-b7ad-ca001a7ee891`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70921::DEBUG::2014-09-25 17:42:17,516::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70921::DEBUG::2014-09-25 17:42:17,516::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70921::DEBUG::2014-09-25 17:42:17,516::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aa743a85-d27d-4fbe-b7ad-ca001a7ee891`::Granted request >Thread-70921::DEBUG::2014-09-25 17:42:17,518::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a73b058d-37ce-4959-a573-3bd95b83c435`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70921::DEBUG::2014-09-25 17:42:17,518::task::993::Storage.TaskManager.Task::(_decref) Task=`a73b058d-37ce-4959-a573-3bd95b83c435`::ref 1 aborting False >Thread-70921::INFO::2014-09-25 17:42:17,523::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70921::DEBUG::2014-09-25 17:42:17,523::task::1191::Storage.TaskManager.Task::(prepare) Task=`a73b058d-37ce-4959-a573-3bd95b83c435`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70921::DEBUG::2014-09-25 17:42:17,523::task::595::Storage.TaskManager.Task::(_updateState) Task=`a73b058d-37ce-4959-a573-3bd95b83c435`::moving from state preparing -> state finished >Thread-70921::DEBUG::2014-09-25 17:42:17,523::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70921::DEBUG::2014-09-25 17:42:17,524::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70921::DEBUG::2014-09-25 17:42:17,524::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70921::DEBUG::2014-09-25 17:42:17,524::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70921::DEBUG::2014-09-25 17:42:17,524::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70921::DEBUG::2014-09-25 17:42:17,525::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70921::DEBUG::2014-09-25 17:42:17,525::task::993::Storage.TaskManager.Task::(_decref) Task=`a73b058d-37ce-4959-a573-3bd95b83c435`::ref 0 aborting False >Thread-70921::DEBUG::2014-09-25 17:42:17,525::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70921::DEBUG::2014-09-25 17:42:17,526::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:18,455::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:18,457::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70922::DEBUG::2014-09-25 17:42:18,459::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:42:18,504::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:18,504::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:18,504::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:18,504::__init__::232::IOProcess::(_processLogs) (109108) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:18,518::__init__::232::IOProcess::(_processLogs) (109108) Queuing response >Thread-79::DEBUG::2014-09-25 17:42:18,519::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:42:18,544::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00594862 s, 94.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:18,545::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:18,546::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:18,546::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:18,546::__init__::232::IOProcess::(_processLogs) (109109) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:18,550::__init__::232::IOProcess::(_processLogs) (109109) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:18,551::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:18,552::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:18,553::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:18,553::__init__::232::IOProcess::(_processLogs) (109110) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:18,553::__init__::232::IOProcess::(_processLogs) (109110) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:18,553::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:18,554::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:18,554::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:18,555::__init__::232::IOProcess::(_processLogs) (109111) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:18,555::__init__::232::IOProcess::(_processLogs) (109111) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:18,556::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:18,556::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:18,557::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:18,557::__init__::232::IOProcess::(_processLogs) (109112) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:18,558::__init__::232::IOProcess::(_processLogs) (109112) Queuing response >Dummy-148::DEBUG::2014-09-25 17:42:19,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:19,394::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211282 s, 48.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:42:21,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:21,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013177 s, 77.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:21,475::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:21,476::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70923::DEBUG::2014-09-25 17:42:21,479::task::595::Storage.TaskManager.Task::(_updateState) Task=`21931808-7d07-4a52-b05e-394615c54cc0`::moving from state init -> state preparing >Thread-70923::INFO::2014-09-25 17:42:21,479::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70923::INFO::2014-09-25 17:42:21,480::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00594862', 'lastCheck': '2.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000531464', 'lastCheck': '5.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00452863', 'lastCheck': '8.5', 'valid': True}} >Thread-70923::DEBUG::2014-09-25 17:42:21,480::task::1191::Storage.TaskManager.Task::(prepare) Task=`21931808-7d07-4a52-b05e-394615c54cc0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00594862', 'lastCheck': '2.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000531464', 'lastCheck': '5.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00452863', 'lastCheck': '8.5', 'valid': True}} >Thread-70923::DEBUG::2014-09-25 17:42:21,480::task::595::Storage.TaskManager.Task::(_updateState) Task=`21931808-7d07-4a52-b05e-394615c54cc0`::moving from state preparing -> state finished >Thread-70923::DEBUG::2014-09-25 17:42:21,480::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70923::DEBUG::2014-09-25 17:42:21,480::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70923::DEBUG::2014-09-25 17:42:21,481::task::993::Storage.TaskManager.Task::(_decref) Task=`21931808-7d07-4a52-b05e-394615c54cc0`::ref 0 aborting False >Thread-70923::DEBUG::2014-09-25 17:42:21,483::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:21,506::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:21,507::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70924::DEBUG::2014-09-25 17:42:21,509::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:42:23,022::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:23,023::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:23,023::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:23,023::__init__::232::IOProcess::(_processLogs) (109113) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:23,026::__init__::232::IOProcess::(_processLogs) (109113) Queuing response >Thread-81::DEBUG::2014-09-25 17:42:23,028::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:42:23,055::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00514159 s, 72.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:23,056::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:23,056::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:23,057::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:23,057::__init__::232::IOProcess::(_processLogs) (109114) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:23,058::__init__::232::IOProcess::(_processLogs) (109114) Queuing response >Dummy-148::DEBUG::2014-09-25 17:42:23,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:23,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175996 s, 58.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:24,562::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:24,563::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70925::DEBUG::2014-09-25 17:42:24,565::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:25,506::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:25,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205491 s, 49.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:25,697::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:25,698::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:25,698::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:25,698::__init__::232::IOProcess::(_processLogs) (109115) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:25,699::__init__::232::IOProcess::(_processLogs) (109115) Queuing response >Thread-80::DEBUG::2014-09-25 17:42:25,700::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:42:25,710::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000429264 s, 881 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:25,711::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:25,712::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:25,712::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:25,712::__init__::232::IOProcess::(_processLogs) (109116) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:25,713::__init__::232::IOProcess::(_processLogs) (109116) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:27,555::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70926::DEBUG::2014-09-25 17:42:27,556::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:42:27,556::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70926::DEBUG::2014-09-25 17:42:27,558::task::595::Storage.TaskManager.Task::(_updateState) Task=`4e8d1ac3-dd93-4bc5-9462-09b5350e7ee1`::moving from state init -> state preparing >Thread-70926::INFO::2014-09-25 17:42:27,559::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70926::INFO::2014-09-25 17:42:27,559::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70926::DEBUG::2014-09-25 17:42:27,560::task::1191::Storage.TaskManager.Task::(prepare) Task=`4e8d1ac3-dd93-4bc5-9462-09b5350e7ee1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70926::DEBUG::2014-09-25 17:42:27,560::task::595::Storage.TaskManager.Task::(_updateState) Task=`4e8d1ac3-dd93-4bc5-9462-09b5350e7ee1`::moving from state preparing -> state finished >Thread-70926::DEBUG::2014-09-25 17:42:27,560::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70926::DEBUG::2014-09-25 17:42:27,561::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Dummy-148::DEBUG::2014-09-25 17:42:27,561::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-70926::DEBUG::2014-09-25 17:42:27,562::task::993::Storage.TaskManager.Task::(_decref) Task=`4e8d1ac3-dd93-4bc5-9462-09b5350e7ee1`::ref 0 aborting False >Thread-70926::DEBUG::2014-09-25 17:42:27,570::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70926::DEBUG::2014-09-25 17:42:27,571::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:27,580::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:27,581::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70927::DEBUG::2014-09-25 17:42:27,583::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:27,591::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:27,593::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70928::DEBUG::2014-09-25 17:42:27,593::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70928::DEBUG::2014-09-25 17:42:27,594::task::595::Storage.TaskManager.Task::(_updateState) Task=`b843aa11-5832-47c4-b389-7c8b82d3f078`::moving from state init -> state preparing >Thread-70928::INFO::2014-09-25 17:42:27,595::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70928::DEBUG::2014-09-25 17:42:27,596::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e7b6d17b-f4c5-49e3-81f2-94928f140ab3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70928::DEBUG::2014-09-25 17:42:27,596::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70928::DEBUG::2014-09-25 17:42:27,596::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70928::DEBUG::2014-09-25 17:42:27,596::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e7b6d17b-f4c5-49e3-81f2-94928f140ab3`::Granted request >Thread-70928::DEBUG::2014-09-25 17:42:27,597::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b843aa11-5832-47c4-b389-7c8b82d3f078`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70928::DEBUG::2014-09-25 17:42:27,597::task::993::Storage.TaskManager.Task::(_decref) Task=`b843aa11-5832-47c4-b389-7c8b82d3f078`::ref 1 aborting False >Thread-70928::INFO::2014-09-25 17:42:27,607::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70928::DEBUG::2014-09-25 17:42:27,609::task::1191::Storage.TaskManager.Task::(prepare) Task=`b843aa11-5832-47c4-b389-7c8b82d3f078`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70928::DEBUG::2014-09-25 17:42:27,611::task::595::Storage.TaskManager.Task::(_updateState) Task=`b843aa11-5832-47c4-b389-7c8b82d3f078`::moving from state preparing -> state finished >Thread-70928::DEBUG::2014-09-25 17:42:27,613::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70928::DEBUG::2014-09-25 17:42:27,614::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70928::DEBUG::2014-09-25 17:42:27,617::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70928::DEBUG::2014-09-25 17:42:27,620::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70928::DEBUG::2014-09-25 17:42:27,621::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70928::DEBUG::2014-09-25 17:42:27,623::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70928::DEBUG::2014-09-25 17:42:27,626::task::993::Storage.TaskManager.Task::(_decref) Task=`b843aa11-5832-47c4-b389-7c8b82d3f078`::ref 0 aborting False >Dummy-148::DEBUG::2014-09-25 17:42:27,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0494896 s, 20.7 MB/s\n'; <rc> = 0 >Thread-70928::DEBUG::2014-09-25 17:42:27,629::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70928::DEBUG::2014-09-25 17:42:27,631::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:42:28,558::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:28,559::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:28,559::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:28,559::__init__::232::IOProcess::(_processLogs) (109117) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:28,561::__init__::232::IOProcess::(_processLogs) (109117) Queuing response >Thread-79::DEBUG::2014-09-25 17:42:28,562::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:42:28,588::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00449093 s, 125 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:28,589::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:28,589::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:28,589::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:28,590::__init__::232::IOProcess::(_processLogs) (109118) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:28,597::__init__::232::IOProcess::(_processLogs) (109118) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:28,598::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:28,599::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:28,599::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:28,599::__init__::232::IOProcess::(_processLogs) (109119) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:28,599::__init__::232::IOProcess::(_processLogs) (109119) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:28,600::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:28,600::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:28,601::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:28,601::__init__::232::IOProcess::(_processLogs) (109120) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:28,601::__init__::232::IOProcess::(_processLogs) (109120) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:28,602::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:28,603::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:28,603::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:28,604::__init__::232::IOProcess::(_processLogs) (109121) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:28,604::__init__::232::IOProcess::(_processLogs) (109121) Queuing response >Dummy-148::DEBUG::2014-09-25 17:42:29,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:29,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213007 s, 48.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:30,599::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:30,600::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70929::DEBUG::2014-09-25 17:42:30,602::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:31,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:31,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0206025 s, 49.7 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 17:42:33,059::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:42:33,064::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:33,064::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:33,064::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:33,065::__init__::232::IOProcess::(_processLogs) (109122) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:42:33,065::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:33,065::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:33,066::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:33,066::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:33,066::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:33,066::__init__::232::IOProcess::(_processLogs) (109123) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:42:33,067::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:33,068::__init__::232::IOProcess::(_processLogs) (109124) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:42:33,068::__init__::232::IOProcess::(_processLogs) (109123) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:33,077::__init__::232::IOProcess::(_processLogs) (109122) Queuing response >Thread-81::DEBUG::2014-09-25 17:42:33,081::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:42:33,082::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:33,082::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:33,082::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:33,083::__init__::232::IOProcess::(_processLogs) (109125) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:42:33,086::__init__::232::IOProcess::(_processLogs) (109124) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:33,122::__init__::232::IOProcess::(_processLogs) (109125) Queuing response >Thread-81::DEBUG::2014-09-25 17:42:33,122::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:42:33,123::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:33,123::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:33,124::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:33,124::__init__::232::IOProcess::(_processLogs) (109126) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:42:33,135::__init__::232::IOProcess::(_processLogs) (109126) Queuing response >Thread-81::DEBUG::2014-09-25 17:42:33,135::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:42:33,137::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:33,137::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:33,137::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:33,138::__init__::232::IOProcess::(_processLogs) (109127) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:33,138::__init__::232::IOProcess::(_processLogs) (109127) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:33,139::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:33,139::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:33,140::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:33,140::__init__::232::IOProcess::(_processLogs) (109128) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:42:33,157::__init__::232::IOProcess::(_processLogs) (109128) Queuing response >Thread-81::DEBUG::2014-09-25 17:42:33,157::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 17:42:33,158::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:42:33,158::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:42:33,159::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:33,160::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:33,160::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:33,160::__init__::232::IOProcess::(_processLogs) (109129) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:33,163::__init__::232::IOProcess::(_processLogs) (109129) Queuing response >Thread-81::DEBUG::2014-09-25 17:42:33,164::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:42:33,195::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0149 s, 24.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:33,196::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:33,196::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:33,196::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:33,197::__init__::232::IOProcess::(_processLogs) (109130) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:33,211::__init__::232::IOProcess::(_processLogs) (109130) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:33,618::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:33,620::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70933::DEBUG::2014-09-25 17:42:33,622::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:33,770::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:33,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170002 s, 60.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:35,715::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:35,716::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:35,716::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:35,716::__init__::232::IOProcess::(_processLogs) (109131) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:35,716::__init__::232::IOProcess::(_processLogs) (109131) Queuing response >Thread-80::DEBUG::2014-09-25 17:42:35,718::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:42:35,726::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000370444 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:35,727::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:35,728::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:35,728::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:35,728::__init__::232::IOProcess::(_processLogs) (109132) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:35,729::__init__::232::IOProcess::(_processLogs) (109132) Queuing response >Dummy-148::DEBUG::2014-09-25 17:42:35,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:35,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209536 s, 48.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:36,643::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:36,646::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70934::DEBUG::2014-09-25 17:42:36,648::task::595::Storage.TaskManager.Task::(_updateState) Task=`99ba4a45-115d-4025-91be-bd54e4580ca3`::moving from state init -> state preparing >Thread-70934::INFO::2014-09-25 17:42:36,649::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70934::INFO::2014-09-25 17:42:36,649::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00449093', 'lastCheck': '8.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000370444', 'lastCheck': '0.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0149', 'lastCheck': '3.4', 'valid': True}} >Thread-70934::DEBUG::2014-09-25 17:42:36,649::task::1191::Storage.TaskManager.Task::(prepare) Task=`99ba4a45-115d-4025-91be-bd54e4580ca3`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00449093', 'lastCheck': '8.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000370444', 'lastCheck': '0.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0149', 'lastCheck': '3.4', 'valid': True}} >Thread-70934::DEBUG::2014-09-25 17:42:36,650::task::595::Storage.TaskManager.Task::(_updateState) Task=`99ba4a45-115d-4025-91be-bd54e4580ca3`::moving from state preparing -> state finished >Thread-70934::DEBUG::2014-09-25 17:42:36,650::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70934::DEBUG::2014-09-25 17:42:36,650::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70934::DEBUG::2014-09-25 17:42:36,650::task::993::Storage.TaskManager.Task::(_decref) Task=`99ba4a45-115d-4025-91be-bd54e4580ca3`::ref 0 aborting False >Thread-70934::DEBUG::2014-09-25 17:42:36,653::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:36,675::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:36,676::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70935::DEBUG::2014-09-25 17:42:36,678::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:37,663::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70936::DEBUG::2014-09-25 17:42:37,664::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:42:37,664::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70936::DEBUG::2014-09-25 17:42:37,667::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc99648e-03d6-4ad8-a8f1-02d57d47d3dc`::moving from state init -> state preparing >Thread-70936::INFO::2014-09-25 17:42:37,667::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70936::INFO::2014-09-25 17:42:37,667::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70936::DEBUG::2014-09-25 17:42:37,668::task::1191::Storage.TaskManager.Task::(prepare) Task=`dc99648e-03d6-4ad8-a8f1-02d57d47d3dc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70936::DEBUG::2014-09-25 17:42:37,668::task::595::Storage.TaskManager.Task::(_updateState) Task=`dc99648e-03d6-4ad8-a8f1-02d57d47d3dc`::moving from state preparing -> state finished >Thread-70936::DEBUG::2014-09-25 17:42:37,668::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70936::DEBUG::2014-09-25 17:42:37,668::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70936::DEBUG::2014-09-25 17:42:37,669::task::993::Storage.TaskManager.Task::(_decref) Task=`dc99648e-03d6-4ad8-a8f1-02d57d47d3dc`::ref 0 aborting False >Thread-70936::DEBUG::2014-09-25 17:42:37,669::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70936::DEBUG::2014-09-25 17:42:37,669::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:37,683::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:37,684::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70937::DEBUG::2014-09-25 17:42:37,684::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70937::DEBUG::2014-09-25 17:42:37,686::task::595::Storage.TaskManager.Task::(_updateState) Task=`5ae3c9b8-ba67-4d2f-ab56-77bddd0ead2f`::moving from state init -> state preparing >Thread-70937::INFO::2014-09-25 17:42:37,686::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70937::DEBUG::2014-09-25 17:42:37,687::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7e933a9f-2a20-46d1-93c6-627b33f89495`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70937::DEBUG::2014-09-25 17:42:37,687::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70937::DEBUG::2014-09-25 17:42:37,688::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70937::DEBUG::2014-09-25 17:42:37,688::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7e933a9f-2a20-46d1-93c6-627b33f89495`::Granted request >Thread-70937::DEBUG::2014-09-25 17:42:37,688::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5ae3c9b8-ba67-4d2f-ab56-77bddd0ead2f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70937::DEBUG::2014-09-25 17:42:37,689::task::993::Storage.TaskManager.Task::(_decref) Task=`5ae3c9b8-ba67-4d2f-ab56-77bddd0ead2f`::ref 1 aborting False >Thread-70937::INFO::2014-09-25 17:42:37,693::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70937::DEBUG::2014-09-25 17:42:37,693::task::1191::Storage.TaskManager.Task::(prepare) Task=`5ae3c9b8-ba67-4d2f-ab56-77bddd0ead2f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70937::DEBUG::2014-09-25 17:42:37,693::task::595::Storage.TaskManager.Task::(_updateState) Task=`5ae3c9b8-ba67-4d2f-ab56-77bddd0ead2f`::moving from state preparing -> state finished >Thread-70937::DEBUG::2014-09-25 17:42:37,694::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70937::DEBUG::2014-09-25 17:42:37,694::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70937::DEBUG::2014-09-25 17:42:37,694::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70937::DEBUG::2014-09-25 17:42:37,695::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70937::DEBUG::2014-09-25 17:42:37,695::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70937::DEBUG::2014-09-25 17:42:37,695::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70937::DEBUG::2014-09-25 17:42:37,695::task::993::Storage.TaskManager.Task::(_decref) Task=`5ae3c9b8-ba67-4d2f-ab56-77bddd0ead2f`::ref 0 aborting False >Thread-70937::DEBUG::2014-09-25 17:42:37,696::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70937::DEBUG::2014-09-25 17:42:37,696::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:37,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:37,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0219108 s, 46.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:38,605::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:38,605::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:38,606::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:38,606::__init__::232::IOProcess::(_processLogs) (109133) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:38,607::__init__::232::IOProcess::(_processLogs) (109133) Queuing response >Thread-79::DEBUG::2014-09-25 17:42:38,608::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:42:38,616::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0003589 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:38,617::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:38,617::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:38,618::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:38,619::__init__::232::IOProcess::(_processLogs) (109134) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:38,620::__init__::232::IOProcess::(_processLogs) (109134) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:38,620::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:38,620::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:38,620::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:38,621::__init__::232::IOProcess::(_processLogs) (109135) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:38,621::__init__::232::IOProcess::(_processLogs) (109135) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:38,622::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:38,622::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:38,623::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:38,623::__init__::232::IOProcess::(_processLogs) (109136) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:38,623::__init__::232::IOProcess::(_processLogs) (109136) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:38,624::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:38,625::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:38,626::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:38,626::__init__::232::IOProcess::(_processLogs) (109137) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:38,626::__init__::232::IOProcess::(_processLogs) (109137) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:39,740::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:39,742::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70938::DEBUG::2014-09-25 17:42:39,744::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:39,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:39,964::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016221 s, 63.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:42:41,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:42,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154438 s, 66.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:42,760::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:42,761::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70939::DEBUG::2014-09-25 17:42:42,763::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:42:43,213::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:43,214::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:43,214::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:43,214::__init__::232::IOProcess::(_processLogs) (109138) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:43,214::__init__::232::IOProcess::(_processLogs) (109138) Queuing response >Thread-81::DEBUG::2014-09-25 17:42:43,216::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:42:43,225::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000416627 s, 890 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:43,226::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:43,226::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:43,227::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:43,228::__init__::232::IOProcess::(_processLogs) (109139) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:43,228::__init__::232::IOProcess::(_processLogs) (109139) Queuing response >Dummy-148::DEBUG::2014-09-25 17:42:44,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:44,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186413 s, 54.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:45,731::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:45,731::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:45,732::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:45,732::__init__::232::IOProcess::(_processLogs) (109140) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:45,732::__init__::232::IOProcess::(_processLogs) (109140) Queuing response >Thread-80::DEBUG::2014-09-25 17:42:45,733::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:42:45,743::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000442026 s, 855 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:45,744::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:45,745::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:45,745::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:45,745::__init__::232::IOProcess::(_processLogs) (109141) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:45,746::__init__::232::IOProcess::(_processLogs) (109141) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:45,778::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:45,779::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70940::DEBUG::2014-09-25 17:42:45,782::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:46,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:46,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0264698 s, 38.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:47,729::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:47,731::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70941::DEBUG::2014-09-25 17:42:47,731::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70941::DEBUG::2014-09-25 17:42:47,733::task::595::Storage.TaskManager.Task::(_updateState) Task=`485406a8-3586-418b-a88d-d1f4f12b1195`::moving from state init -> state preparing >Thread-70941::INFO::2014-09-25 17:42:47,733::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70941::INFO::2014-09-25 17:42:47,734::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70941::DEBUG::2014-09-25 17:42:47,734::task::1191::Storage.TaskManager.Task::(prepare) Task=`485406a8-3586-418b-a88d-d1f4f12b1195`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70941::DEBUG::2014-09-25 17:42:47,734::task::595::Storage.TaskManager.Task::(_updateState) Task=`485406a8-3586-418b-a88d-d1f4f12b1195`::moving from state preparing -> state finished >Thread-70941::DEBUG::2014-09-25 17:42:47,734::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70941::DEBUG::2014-09-25 17:42:47,734::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70941::DEBUG::2014-09-25 17:42:47,735::task::993::Storage.TaskManager.Task::(_decref) Task=`485406a8-3586-418b-a88d-d1f4f12b1195`::ref 0 aborting False >Thread-70941::DEBUG::2014-09-25 17:42:47,735::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70941::DEBUG::2014-09-25 17:42:47,735::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:47,749::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:47,750::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70942::DEBUG::2014-09-25 17:42:47,750::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70942::DEBUG::2014-09-25 17:42:47,752::task::595::Storage.TaskManager.Task::(_updateState) Task=`b8659578-e23a-4f31-b44e-ca5fa7a410fc`::moving from state init -> state preparing >Thread-70942::INFO::2014-09-25 17:42:47,752::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70942::DEBUG::2014-09-25 17:42:47,753::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a1df37ba-6690-4d57-91eb-fec84fa40b62`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70942::DEBUG::2014-09-25 17:42:47,753::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70942::DEBUG::2014-09-25 17:42:47,754::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70942::DEBUG::2014-09-25 17:42:47,754::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a1df37ba-6690-4d57-91eb-fec84fa40b62`::Granted request >Thread-70942::DEBUG::2014-09-25 17:42:47,754::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b8659578-e23a-4f31-b44e-ca5fa7a410fc`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70942::DEBUG::2014-09-25 17:42:47,755::task::993::Storage.TaskManager.Task::(_decref) Task=`b8659578-e23a-4f31-b44e-ca5fa7a410fc`::ref 1 aborting False >Thread-70942::INFO::2014-09-25 17:42:47,759::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70942::DEBUG::2014-09-25 17:42:47,759::task::1191::Storage.TaskManager.Task::(prepare) Task=`b8659578-e23a-4f31-b44e-ca5fa7a410fc`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70942::DEBUG::2014-09-25 17:42:47,759::task::595::Storage.TaskManager.Task::(_updateState) Task=`b8659578-e23a-4f31-b44e-ca5fa7a410fc`::moving from state preparing -> state finished >Thread-70942::DEBUG::2014-09-25 17:42:47,760::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70942::DEBUG::2014-09-25 17:42:47,760::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70942::DEBUG::2014-09-25 17:42:47,760::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70942::DEBUG::2014-09-25 17:42:47,761::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70942::DEBUG::2014-09-25 17:42:47,761::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70942::DEBUG::2014-09-25 17:42:47,761::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70942::DEBUG::2014-09-25 17:42:47,761::task::993::Storage.TaskManager.Task::(_decref) Task=`b8659578-e23a-4f31-b44e-ca5fa7a410fc`::ref 0 aborting False >Thread-70942::DEBUG::2014-09-25 17:42:47,762::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70942::DEBUG::2014-09-25 17:42:47,762::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:48,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:48,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164514 s, 62.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:48,627::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:48,627::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:48,627::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:48,628::__init__::232::IOProcess::(_processLogs) (109142) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:48,634::__init__::232::IOProcess::(_processLogs) (109142) Queuing response >Thread-79::DEBUG::2014-09-25 17:42:48,635::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:42:48,651::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00436139 s, 129 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:48,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:48,653::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:48,653::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:48,653::__init__::232::IOProcess::(_processLogs) (109143) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:48,656::__init__::232::IOProcess::(_processLogs) (109143) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:48,658::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:48,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:48,658::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:48,658::__init__::232::IOProcess::(_processLogs) (109144) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:48,660::__init__::232::IOProcess::(_processLogs) (109144) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:48,661::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:48,661::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:48,662::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:48,662::__init__::232::IOProcess::(_processLogs) (109145) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:48,666::__init__::232::IOProcess::(_processLogs) (109145) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:48,668::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:48,668::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:48,668::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:48,668::__init__::232::IOProcess::(_processLogs) (109146) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:48,672::__init__::232::IOProcess::(_processLogs) (109146) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:48,798::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:48,799::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70943::DEBUG::2014-09-25 17:42:48,801::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:50,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:50,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014745 s, 69.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:51,816::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:51,817::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70944::DEBUG::2014-09-25 17:42:51,820::task::595::Storage.TaskManager.Task::(_updateState) Task=`32e9e622-3e08-481f-b6aa-b70e3ee67901`::moving from state init -> state preparing >Thread-70944::INFO::2014-09-25 17:42:51,820::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70944::INFO::2014-09-25 17:42:51,821::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00436139', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000442026', 'lastCheck': '6.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000416627', 'lastCheck': '8.6', 'valid': True}} >Thread-70944::DEBUG::2014-09-25 17:42:51,821::task::1191::Storage.TaskManager.Task::(prepare) Task=`32e9e622-3e08-481f-b6aa-b70e3ee67901`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00436139', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000442026', 'lastCheck': '6.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000416627', 'lastCheck': '8.6', 'valid': True}} >Thread-70944::DEBUG::2014-09-25 17:42:51,821::task::595::Storage.TaskManager.Task::(_updateState) Task=`32e9e622-3e08-481f-b6aa-b70e3ee67901`::moving from state preparing -> state finished >Thread-70944::DEBUG::2014-09-25 17:42:51,821::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70944::DEBUG::2014-09-25 17:42:51,821::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70944::DEBUG::2014-09-25 17:42:51,822::task::993::Storage.TaskManager.Task::(_decref) Task=`32e9e622-3e08-481f-b6aa-b70e3ee67901`::ref 0 aborting False >Thread-70944::DEBUG::2014-09-25 17:42:51,824::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:51,844::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:51,845::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70945::DEBUG::2014-09-25 17:42:51,847::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:52,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:52,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014604 s, 70.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:53,229::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:53,229::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:53,230::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:53,230::__init__::232::IOProcess::(_processLogs) (109147) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:53,230::__init__::232::IOProcess::(_processLogs) (109147) Queuing response >Thread-81::DEBUG::2014-09-25 17:42:53,232::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:42:53,250::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0057827 s, 64.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:53,251::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:53,251::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:53,252::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:53,252::__init__::232::IOProcess::(_processLogs) (109148) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:53,257::__init__::232::IOProcess::(_processLogs) (109148) Queuing response >Dummy-148::DEBUG::2014-09-25 17:42:54,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:54,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175404 s, 58.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:54,904::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:54,906::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70946::DEBUG::2014-09-25 17:42:54,908::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:42:55,748::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:55,748::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:55,749::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:55,749::__init__::232::IOProcess::(_processLogs) (109149) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:55,749::__init__::232::IOProcess::(_processLogs) (109149) Queuing response >Thread-80::DEBUG::2014-09-25 17:42:55,750::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:42:55,759::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000446198 s, 847 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:55,760::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:55,761::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:55,761::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:55,763::__init__::232::IOProcess::(_processLogs) (109150) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:55,763::__init__::232::IOProcess::(_processLogs) (109150) Queuing response >Dummy-148::DEBUG::2014-09-25 17:42:56,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:56,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130934 s, 78.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:57,790::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:57,791::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70947::DEBUG::2014-09-25 17:42:57,791::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70947::DEBUG::2014-09-25 17:42:57,793::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c5052f3-c12b-4f30-b05d-5d2d3272fbf8`::moving from state init -> state preparing >Thread-70947::INFO::2014-09-25 17:42:57,794::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70947::INFO::2014-09-25 17:42:57,794::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70947::DEBUG::2014-09-25 17:42:57,794::task::1191::Storage.TaskManager.Task::(prepare) Task=`0c5052f3-c12b-4f30-b05d-5d2d3272fbf8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70947::DEBUG::2014-09-25 17:42:57,795::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c5052f3-c12b-4f30-b05d-5d2d3272fbf8`::moving from state preparing -> state finished >Thread-70947::DEBUG::2014-09-25 17:42:57,795::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70947::DEBUG::2014-09-25 17:42:57,795::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70947::DEBUG::2014-09-25 17:42:57,795::task::993::Storage.TaskManager.Task::(_decref) Task=`0c5052f3-c12b-4f30-b05d-5d2d3272fbf8`::ref 0 aborting False >Thread-70947::DEBUG::2014-09-25 17:42:57,796::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70947::DEBUG::2014-09-25 17:42:57,796::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:57,810::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:57,812::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70948::DEBUG::2014-09-25 17:42:57,812::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70948::DEBUG::2014-09-25 17:42:57,813::task::595::Storage.TaskManager.Task::(_updateState) Task=`5c109552-0c06-427e-8231-627dff02c24c`::moving from state init -> state preparing >Thread-70948::INFO::2014-09-25 17:42:57,814::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70948::DEBUG::2014-09-25 17:42:57,815::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`16f53d01-9dc0-4544-aa88-662053869c82`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70948::DEBUG::2014-09-25 17:42:57,816::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70948::DEBUG::2014-09-25 17:42:57,816::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70948::DEBUG::2014-09-25 17:42:57,816::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`16f53d01-9dc0-4544-aa88-662053869c82`::Granted request >Thread-70948::DEBUG::2014-09-25 17:42:57,817::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5c109552-0c06-427e-8231-627dff02c24c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70948::DEBUG::2014-09-25 17:42:57,817::task::993::Storage.TaskManager.Task::(_decref) Task=`5c109552-0c06-427e-8231-627dff02c24c`::ref 1 aborting False >Thread-70948::INFO::2014-09-25 17:42:57,821::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70948::DEBUG::2014-09-25 17:42:57,822::task::1191::Storage.TaskManager.Task::(prepare) Task=`5c109552-0c06-427e-8231-627dff02c24c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70948::DEBUG::2014-09-25 17:42:57,822::task::595::Storage.TaskManager.Task::(_updateState) Task=`5c109552-0c06-427e-8231-627dff02c24c`::moving from state preparing -> state finished >Thread-70948::DEBUG::2014-09-25 17:42:57,822::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70948::DEBUG::2014-09-25 17:42:57,822::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70948::DEBUG::2014-09-25 17:42:57,823::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70948::DEBUG::2014-09-25 17:42:57,823::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70948::DEBUG::2014-09-25 17:42:57,823::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70948::DEBUG::2014-09-25 17:42:57,824::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70948::DEBUG::2014-09-25 17:42:57,824::task::993::Storage.TaskManager.Task::(_decref) Task=`5c109552-0c06-427e-8231-627dff02c24c`::ref 0 aborting False >Thread-70948::DEBUG::2014-09-25 17:42:57,824::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70948::DEBUG::2014-09-25 17:42:57,825::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:42:57,926::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:42:57,927::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70949::DEBUG::2014-09-25 17:42:57,929::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:42:58,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:42:58,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181473 s, 56.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:58,674::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:58,674::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:58,674::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:58,675::__init__::232::IOProcess::(_processLogs) (109151) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:58,678::__init__::232::IOProcess::(_processLogs) (109151) Queuing response >Thread-79::DEBUG::2014-09-25 17:42:58,679::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:42:58,696::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00303416 s, 185 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:42:58,697::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:58,697::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:58,697::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:58,697::__init__::232::IOProcess::(_processLogs) (109152) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:42:58,700::__init__::232::IOProcess::(_processLogs) (109152) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:58,701::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:58,702::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:58,702::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:58,703::__init__::232::IOProcess::(_processLogs) (109153) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:58,703::__init__::232::IOProcess::(_processLogs) (109153) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:58,703::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:58,704::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:58,704::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:58,704::__init__::232::IOProcess::(_processLogs) (109154) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:58,704::__init__::232::IOProcess::(_processLogs) (109154) Queuing response >Thread-29::DEBUG::2014-09-25 17:42:58,705::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:42:58,705::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:42:58,706::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:42:58,707::__init__::232::IOProcess::(_processLogs) (109155) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:42:58,707::__init__::232::IOProcess::(_processLogs) (109155) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:00,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:00,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160227 s, 63.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:00,944::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:00,945::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70950::DEBUG::2014-09-25 17:43:00,948::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:02,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:02,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133859 s, 76.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:03,259::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:03,260::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:03,260::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:03,260::__init__::232::IOProcess::(_processLogs) (109156) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:03,274::__init__::232::IOProcess::(_processLogs) (109156) Queuing response >Thread-81::DEBUG::2014-09-25 17:43:03,277::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:43:03,297::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00447871 s, 82.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:03,299::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:03,299::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:03,299::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:03,299::__init__::232::IOProcess::(_processLogs) (109157) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:03,303::__init__::232::IOProcess::(_processLogs) (109157) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:03,962::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:03,964::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70951::DEBUG::2014-09-25 17:43:03,966::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:04,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:04,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0229973 s, 44.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:05,764::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:05,764::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:05,765::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:05,765::__init__::232::IOProcess::(_processLogs) (109158) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:05,765::__init__::232::IOProcess::(_processLogs) (109158) Queuing response >Thread-80::DEBUG::2014-09-25 17:43:05,766::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:43:05,775::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000398452 s, 949 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:05,777::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:05,777::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:05,777::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:05,777::__init__::232::IOProcess::(_processLogs) (109159) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:05,778::__init__::232::IOProcess::(_processLogs) (109159) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:06,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:06,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186638 s, 54.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:06,980::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:06,981::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70952::DEBUG::2014-09-25 17:43:06,984::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcec94c2-b6ba-4d2d-8355-75c93749e71b`::moving from state init -> state preparing >Thread-70952::INFO::2014-09-25 17:43:06,984::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70952::INFO::2014-09-25 17:43:06,984::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00303416', 'lastCheck': '8.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000398452', 'lastCheck': '1.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00447871', 'lastCheck': '3.7', 'valid': True}} >Thread-70952::DEBUG::2014-09-25 17:43:06,985::task::1191::Storage.TaskManager.Task::(prepare) Task=`fcec94c2-b6ba-4d2d-8355-75c93749e71b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00303416', 'lastCheck': '8.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000398452', 'lastCheck': '1.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00447871', 'lastCheck': '3.7', 'valid': True}} >Thread-70952::DEBUG::2014-09-25 17:43:06,985::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcec94c2-b6ba-4d2d-8355-75c93749e71b`::moving from state preparing -> state finished >Thread-70952::DEBUG::2014-09-25 17:43:06,985::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70952::DEBUG::2014-09-25 17:43:06,985::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70952::DEBUG::2014-09-25 17:43:06,986::task::993::Storage.TaskManager.Task::(_decref) Task=`fcec94c2-b6ba-4d2d-8355-75c93749e71b`::ref 0 aborting False >Thread-70952::DEBUG::2014-09-25 17:43:06,988::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:07,009::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:07,011::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70953::DEBUG::2014-09-25 17:43:07,013::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:07,854::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:07,856::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70954::DEBUG::2014-09-25 17:43:07,856::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70954::DEBUG::2014-09-25 17:43:07,858::task::595::Storage.TaskManager.Task::(_updateState) Task=`da4b99eb-8f7e-4934-8829-a1552d83c3da`::moving from state init -> state preparing >Thread-70954::INFO::2014-09-25 17:43:07,858::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70954::INFO::2014-09-25 17:43:07,858::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70954::DEBUG::2014-09-25 17:43:07,859::task::1191::Storage.TaskManager.Task::(prepare) Task=`da4b99eb-8f7e-4934-8829-a1552d83c3da`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70954::DEBUG::2014-09-25 17:43:07,859::task::595::Storage.TaskManager.Task::(_updateState) Task=`da4b99eb-8f7e-4934-8829-a1552d83c3da`::moving from state preparing -> state finished >Thread-70954::DEBUG::2014-09-25 17:43:07,859::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70954::DEBUG::2014-09-25 17:43:07,859::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70954::DEBUG::2014-09-25 17:43:07,860::task::993::Storage.TaskManager.Task::(_decref) Task=`da4b99eb-8f7e-4934-8829-a1552d83c3da`::ref 0 aborting False >Thread-70954::DEBUG::2014-09-25 17:43:07,860::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70954::DEBUG::2014-09-25 17:43:07,860::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:07,874::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:07,875::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70955::DEBUG::2014-09-25 17:43:07,875::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70955::DEBUG::2014-09-25 17:43:07,877::task::595::Storage.TaskManager.Task::(_updateState) Task=`5902fd04-60d6-49fe-b47b-730bdf650f63`::moving from state init -> state preparing >Thread-70955::INFO::2014-09-25 17:43:07,877::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70955::DEBUG::2014-09-25 17:43:07,878::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8b6715d3-9c60-4cf7-964f-b8ec368fd3c7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70955::DEBUG::2014-09-25 17:43:07,878::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70955::DEBUG::2014-09-25 17:43:07,879::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70955::DEBUG::2014-09-25 17:43:07,879::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8b6715d3-9c60-4cf7-964f-b8ec368fd3c7`::Granted request >Thread-70955::DEBUG::2014-09-25 17:43:07,879::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5902fd04-60d6-49fe-b47b-730bdf650f63`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70955::DEBUG::2014-09-25 17:43:07,880::task::993::Storage.TaskManager.Task::(_decref) Task=`5902fd04-60d6-49fe-b47b-730bdf650f63`::ref 1 aborting False >Thread-70955::INFO::2014-09-25 17:43:07,884::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70955::DEBUG::2014-09-25 17:43:07,884::task::1191::Storage.TaskManager.Task::(prepare) Task=`5902fd04-60d6-49fe-b47b-730bdf650f63`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70955::DEBUG::2014-09-25 17:43:07,884::task::595::Storage.TaskManager.Task::(_updateState) Task=`5902fd04-60d6-49fe-b47b-730bdf650f63`::moving from state preparing -> state finished >Thread-70955::DEBUG::2014-09-25 17:43:07,885::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70955::DEBUG::2014-09-25 17:43:07,885::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70955::DEBUG::2014-09-25 17:43:07,885::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70955::DEBUG::2014-09-25 17:43:07,886::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70955::DEBUG::2014-09-25 17:43:07,886::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70955::DEBUG::2014-09-25 17:43:07,886::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70955::DEBUG::2014-09-25 17:43:07,886::task::993::Storage.TaskManager.Task::(_decref) Task=`5902fd04-60d6-49fe-b47b-730bdf650f63`::ref 0 aborting False >Thread-70955::DEBUG::2014-09-25 17:43:07,887::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70955::DEBUG::2014-09-25 17:43:07,887::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:43:08,707::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:08,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:08,708::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:08,708::__init__::232::IOProcess::(_processLogs) (109160) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:08,709::__init__::232::IOProcess::(_processLogs) (109160) Queuing response >Thread-79::DEBUG::2014-09-25 17:43:08,710::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:43:08,728::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00352652 s, 159 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:08,730::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:08,730::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:08,730::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:08,731::__init__::232::IOProcess::(_processLogs) (109161) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:08,734::__init__::232::IOProcess::(_processLogs) (109161) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:08,735::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:08,735::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:08,736::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:08,736::__init__::232::IOProcess::(_processLogs) (109162) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:08,736::__init__::232::IOProcess::(_processLogs) (109162) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:08,737::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:08,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:08,737::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:08,738::__init__::232::IOProcess::(_processLogs) (109163) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:08,738::__init__::232::IOProcess::(_processLogs) (109163) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:08,739::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:08,739::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:08,739::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:08,740::__init__::232::IOProcess::(_processLogs) (109164) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:08,740::__init__::232::IOProcess::(_processLogs) (109164) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:08,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:08,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178776 s, 57.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:10,079::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:10,081::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70956::DEBUG::2014-09-25 17:43:10,083::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:10,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:10,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191039 s, 53.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:43:12,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:12,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0353901 s, 28.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:13,099::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:13,100::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70957::DEBUG::2014-09-25 17:43:13,102::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:43:13,305::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:13,306::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:13,306::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:13,306::__init__::232::IOProcess::(_processLogs) (109165) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:13,310::__init__::232::IOProcess::(_processLogs) (109165) Queuing response >Thread-81::DEBUG::2014-09-25 17:43:13,311::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:43:13,323::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00121783 s, 305 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:13,324::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:13,324::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:13,325::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:13,325::__init__::232::IOProcess::(_processLogs) (109166) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:13,327::__init__::232::IOProcess::(_processLogs) (109166) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:14,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:14,999::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0341634 s, 30.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:15,780::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:15,781::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:15,781::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:15,781::__init__::232::IOProcess::(_processLogs) (109167) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:15,782::__init__::232::IOProcess::(_processLogs) (109167) Queuing response >Thread-80::DEBUG::2014-09-25 17:43:15,783::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:43:15,792::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000497021 s, 761 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:15,793::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:15,793::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:15,794::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:15,796::__init__::232::IOProcess::(_processLogs) (109168) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:15,796::__init__::232::IOProcess::(_processLogs) (109168) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:16,117::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:16,121::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70958::DEBUG::2014-09-25 17:43:16,129::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:17,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:17,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163351 s, 62.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:17,918::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:17,920::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70959::DEBUG::2014-09-25 17:43:17,920::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70959::DEBUG::2014-09-25 17:43:17,922::task::595::Storage.TaskManager.Task::(_updateState) Task=`30ef266c-318e-4c28-a782-2c7422c67a01`::moving from state init -> state preparing >Thread-70959::INFO::2014-09-25 17:43:17,922::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70959::INFO::2014-09-25 17:43:17,923::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70959::DEBUG::2014-09-25 17:43:17,923::task::1191::Storage.TaskManager.Task::(prepare) Task=`30ef266c-318e-4c28-a782-2c7422c67a01`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70959::DEBUG::2014-09-25 17:43:17,923::task::595::Storage.TaskManager.Task::(_updateState) Task=`30ef266c-318e-4c28-a782-2c7422c67a01`::moving from state preparing -> state finished >Thread-70959::DEBUG::2014-09-25 17:43:17,923::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70959::DEBUG::2014-09-25 17:43:17,924::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70959::DEBUG::2014-09-25 17:43:17,924::task::993::Storage.TaskManager.Task::(_decref) Task=`30ef266c-318e-4c28-a782-2c7422c67a01`::ref 0 aborting False >Thread-70959::DEBUG::2014-09-25 17:43:17,924::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70959::DEBUG::2014-09-25 17:43:17,924::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:17,937::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:17,938::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70960::DEBUG::2014-09-25 17:43:17,938::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70960::DEBUG::2014-09-25 17:43:17,940::task::595::Storage.TaskManager.Task::(_updateState) Task=`31564282-4f20-42bc-90a0-478304ac1a73`::moving from state init -> state preparing >Thread-70960::INFO::2014-09-25 17:43:17,940::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70960::DEBUG::2014-09-25 17:43:17,941::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`31673ec4-314a-46c7-bb20-84e646ee0a31`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70960::DEBUG::2014-09-25 17:43:17,941::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70960::DEBUG::2014-09-25 17:43:17,941::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70960::DEBUG::2014-09-25 17:43:17,942::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`31673ec4-314a-46c7-bb20-84e646ee0a31`::Granted request >Thread-70960::DEBUG::2014-09-25 17:43:17,942::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`31564282-4f20-42bc-90a0-478304ac1a73`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70960::DEBUG::2014-09-25 17:43:17,942::task::993::Storage.TaskManager.Task::(_decref) Task=`31564282-4f20-42bc-90a0-478304ac1a73`::ref 1 aborting False >Thread-70960::INFO::2014-09-25 17:43:17,947::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70960::DEBUG::2014-09-25 17:43:17,947::task::1191::Storage.TaskManager.Task::(prepare) Task=`31564282-4f20-42bc-90a0-478304ac1a73`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70960::DEBUG::2014-09-25 17:43:17,947::task::595::Storage.TaskManager.Task::(_updateState) Task=`31564282-4f20-42bc-90a0-478304ac1a73`::moving from state preparing -> state finished >Thread-70960::DEBUG::2014-09-25 17:43:17,948::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70960::DEBUG::2014-09-25 17:43:17,948::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70960::DEBUG::2014-09-25 17:43:17,948::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70960::DEBUG::2014-09-25 17:43:17,948::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70960::DEBUG::2014-09-25 17:43:17,949::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70960::DEBUG::2014-09-25 17:43:17,949::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70960::DEBUG::2014-09-25 17:43:17,949::task::993::Storage.TaskManager.Task::(_decref) Task=`31564282-4f20-42bc-90a0-478304ac1a73`::ref 0 aborting False >Thread-70960::DEBUG::2014-09-25 17:43:17,950::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70960::DEBUG::2014-09-25 17:43:17,950::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:43:18,741::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:18,742::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:18,742::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:18,742::__init__::232::IOProcess::(_processLogs) (109169) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:18,743::__init__::232::IOProcess::(_processLogs) (109169) Queuing response >Thread-79::DEBUG::2014-09-25 17:43:18,744::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:43:18,753::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00035963 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:18,754::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:18,754::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:18,755::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:18,755::__init__::232::IOProcess::(_processLogs) (109170) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:18,755::__init__::232::IOProcess::(_processLogs) (109170) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:18,757::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:18,757::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:18,757::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:18,757::__init__::232::IOProcess::(_processLogs) (109171) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:18,758::__init__::232::IOProcess::(_processLogs) (109171) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:18,758::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:18,759::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:18,759::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:18,760::__init__::232::IOProcess::(_processLogs) (109172) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:18,760::__init__::232::IOProcess::(_processLogs) (109172) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:18,760::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:18,761::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:18,761::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:18,761::__init__::232::IOProcess::(_processLogs) (109173) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:18,761::__init__::232::IOProcess::(_processLogs) (109173) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:19,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:19,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191361 s, 53.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:19,149::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:19,151::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70961::DEBUG::2014-09-25 17:43:19,153::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:21,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:21,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138807 s, 73.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:22,169::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:22,171::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70962::DEBUG::2014-09-25 17:43:22,173::task::595::Storage.TaskManager.Task::(_updateState) Task=`f635e760-1040-44d1-8848-9dc94a65a061`::moving from state init -> state preparing >Thread-70962::INFO::2014-09-25 17:43:22,173::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70962::INFO::2014-09-25 17:43:22,174::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00035963', 'lastCheck': '3.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000497021', 'lastCheck': '6.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00121783', 'lastCheck': '8.8', 'valid': True}} >Thread-70962::DEBUG::2014-09-25 17:43:22,174::task::1191::Storage.TaskManager.Task::(prepare) Task=`f635e760-1040-44d1-8848-9dc94a65a061`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00035963', 'lastCheck': '3.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000497021', 'lastCheck': '6.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00121783', 'lastCheck': '8.8', 'valid': True}} >Thread-70962::DEBUG::2014-09-25 17:43:22,174::task::595::Storage.TaskManager.Task::(_updateState) Task=`f635e760-1040-44d1-8848-9dc94a65a061`::moving from state preparing -> state finished >Thread-70962::DEBUG::2014-09-25 17:43:22,175::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70962::DEBUG::2014-09-25 17:43:22,175::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70962::DEBUG::2014-09-25 17:43:22,175::task::993::Storage.TaskManager.Task::(_decref) Task=`f635e760-1040-44d1-8848-9dc94a65a061`::ref 0 aborting False >Thread-70962::DEBUG::2014-09-25 17:43:22,177::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:22,199::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:22,199::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70963::DEBUG::2014-09-25 17:43:22,201::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:23,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:23,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0252618 s, 40.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:23,329::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:23,329::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:23,330::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:23,330::__init__::232::IOProcess::(_processLogs) (109174) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:23,358::__init__::232::IOProcess::(_processLogs) (109174) Queuing response >Thread-81::DEBUG::2014-09-25 17:43:23,358::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:43:23,379::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00375801 s, 98.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:23,380::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:23,380::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:23,381::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:23,381::__init__::232::IOProcess::(_processLogs) (109175) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:23,388::__init__::232::IOProcess::(_processLogs) (109175) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:25,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:25,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142867 s, 71.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:25,258::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:25,260::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70964::DEBUG::2014-09-25 17:43:25,262::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:43:25,796::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:25,797::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:25,797::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:25,797::__init__::232::IOProcess::(_processLogs) (109176) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:25,798::__init__::232::IOProcess::(_processLogs) (109176) Queuing response >Thread-80::DEBUG::2014-09-25 17:43:25,799::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:43:25,808::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00038008 s, 995 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:25,809::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:25,809::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:25,809::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:25,810::__init__::232::IOProcess::(_processLogs) (109177) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:25,810::__init__::232::IOProcess::(_processLogs) (109177) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:27,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:27,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162891 s, 62.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:27,981::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:27,983::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70965::DEBUG::2014-09-25 17:43:27,983::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70965::DEBUG::2014-09-25 17:43:27,985::task::595::Storage.TaskManager.Task::(_updateState) Task=`83362934-d43c-4894-ab98-f06c2cbf1437`::moving from state init -> state preparing >Thread-70965::INFO::2014-09-25 17:43:27,985::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70965::INFO::2014-09-25 17:43:27,986::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70965::DEBUG::2014-09-25 17:43:27,986::task::1191::Storage.TaskManager.Task::(prepare) Task=`83362934-d43c-4894-ab98-f06c2cbf1437`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70965::DEBUG::2014-09-25 17:43:27,986::task::595::Storage.TaskManager.Task::(_updateState) Task=`83362934-d43c-4894-ab98-f06c2cbf1437`::moving from state preparing -> state finished >Thread-70965::DEBUG::2014-09-25 17:43:27,986::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70965::DEBUG::2014-09-25 17:43:27,987::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70965::DEBUG::2014-09-25 17:43:27,987::task::993::Storage.TaskManager.Task::(_decref) Task=`83362934-d43c-4894-ab98-f06c2cbf1437`::ref 0 aborting False >Thread-70965::DEBUG::2014-09-25 17:43:27,987::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70965::DEBUG::2014-09-25 17:43:27,988::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:28,006::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:28,007::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70966::DEBUG::2014-09-25 17:43:28,007::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70966::DEBUG::2014-09-25 17:43:28,009::task::595::Storage.TaskManager.Task::(_updateState) Task=`2612841f-a2d8-48c1-86d4-bbbf0df707fc`::moving from state init -> state preparing >Thread-70966::INFO::2014-09-25 17:43:28,009::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70966::DEBUG::2014-09-25 17:43:28,010::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ea038462-c836-459b-b744-44c87f20a5cb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70966::DEBUG::2014-09-25 17:43:28,010::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70966::DEBUG::2014-09-25 17:43:28,011::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70966::DEBUG::2014-09-25 17:43:28,011::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ea038462-c836-459b-b744-44c87f20a5cb`::Granted request >Thread-70966::DEBUG::2014-09-25 17:43:28,011::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`2612841f-a2d8-48c1-86d4-bbbf0df707fc`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70966::DEBUG::2014-09-25 17:43:28,012::task::993::Storage.TaskManager.Task::(_decref) Task=`2612841f-a2d8-48c1-86d4-bbbf0df707fc`::ref 1 aborting False >Thread-70966::INFO::2014-09-25 17:43:28,016::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70966::DEBUG::2014-09-25 17:43:28,016::task::1191::Storage.TaskManager.Task::(prepare) Task=`2612841f-a2d8-48c1-86d4-bbbf0df707fc`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70966::DEBUG::2014-09-25 17:43:28,017::task::595::Storage.TaskManager.Task::(_updateState) Task=`2612841f-a2d8-48c1-86d4-bbbf0df707fc`::moving from state preparing -> state finished >Thread-70966::DEBUG::2014-09-25 17:43:28,017::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70966::DEBUG::2014-09-25 17:43:28,017::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70966::DEBUG::2014-09-25 17:43:28,017::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70966::DEBUG::2014-09-25 17:43:28,018::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70966::DEBUG::2014-09-25 17:43:28,018::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70966::DEBUG::2014-09-25 17:43:28,018::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70966::DEBUG::2014-09-25 17:43:28,019::task::993::Storage.TaskManager.Task::(_decref) Task=`2612841f-a2d8-48c1-86d4-bbbf0df707fc`::ref 0 aborting False >Thread-70966::DEBUG::2014-09-25 17:43:28,019::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70966::DEBUG::2014-09-25 17:43:28,019::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:28,280::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:28,282::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70967::DEBUG::2014-09-25 17:43:28,284::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:43:28,763::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:28,764::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-79::DEBUG::2014-09-25 17:43:28,765::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:43:28,766::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:28,773::__init__::232::IOProcess::(_processLogs) (109178) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:28,775::__init__::232::IOProcess::(_processLogs) (109178) Queuing response >Thread-79::DEBUG::2014-09-25 17:43:28,775::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00035687 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:28,777::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:28,777::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:28,777::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:28,777::__init__::232::IOProcess::(_processLogs) (109179) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:28,778::__init__::232::IOProcess::(_processLogs) (109179) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:28,779::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:28,779::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:28,780::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:28,781::__init__::232::IOProcess::(_processLogs) (109180) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:28,781::__init__::232::IOProcess::(_processLogs) (109180) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:28,781::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:28,781::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:28,782::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:28,783::__init__::232::IOProcess::(_processLogs) (109181) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:28,783::__init__::232::IOProcess::(_processLogs) (109181) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:28,783::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:28,784::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:28,785::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:28,786::__init__::232::IOProcess::(_processLogs) (109182) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:28,786::__init__::232::IOProcess::(_processLogs) (109182) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:29,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:29,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165618 s, 61.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:31,300::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:31,302::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70968::DEBUG::2014-09-25 17:43:31,304::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:31,350::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:31,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.222055 s, 4.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:33,390::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:33,390::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:33,390::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:33,390::__init__::232::IOProcess::(_processLogs) (109183) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:33,397::__init__::232::IOProcess::(_processLogs) (109183) Queuing response >Thread-81::DEBUG::2014-09-25 17:43:33,398::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:43:33,412::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00317843 s, 117 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:33,413::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:33,413::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:33,414::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:33,414::__init__::232::IOProcess::(_processLogs) (109184) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:33,416::__init__::232::IOProcess::(_processLogs) (109184) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:33,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:33,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170095 s, 60.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:34,317::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:34,319::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70969::DEBUG::2014-09-25 17:43:34,321::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:35,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:35,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0187562 s, 54.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:35,813::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:35,813::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:35,813::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:35,813::__init__::232::IOProcess::(_processLogs) (109185) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:35,814::__init__::232::IOProcess::(_processLogs) (109185) Queuing response >Thread-80::DEBUG::2014-09-25 17:43:35,815::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:43:35,824::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000380765 s, 993 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:35,826::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:35,826::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:35,826::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:35,826::__init__::232::IOProcess::(_processLogs) (109186) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:35,827::__init__::232::IOProcess::(_processLogs) (109186) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:37,335::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:37,337::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70970::DEBUG::2014-09-25 17:43:37,339::task::595::Storage.TaskManager.Task::(_updateState) Task=`1bef0965-48d2-45ec-8d3e-2bd78053a02b`::moving from state init -> state preparing >Thread-70970::INFO::2014-09-25 17:43:37,339::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70970::INFO::2014-09-25 17:43:37,340::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00035687', 'lastCheck': '8.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000380765', 'lastCheck': '1.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00317843', 'lastCheck': '3.9', 'valid': True}} >Thread-70970::DEBUG::2014-09-25 17:43:37,340::task::1191::Storage.TaskManager.Task::(prepare) Task=`1bef0965-48d2-45ec-8d3e-2bd78053a02b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00035687', 'lastCheck': '8.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000380765', 'lastCheck': '1.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00317843', 'lastCheck': '3.9', 'valid': True}} >Thread-70970::DEBUG::2014-09-25 17:43:37,340::task::595::Storage.TaskManager.Task::(_updateState) Task=`1bef0965-48d2-45ec-8d3e-2bd78053a02b`::moving from state preparing -> state finished >Thread-70970::DEBUG::2014-09-25 17:43:37,341::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70970::DEBUG::2014-09-25 17:43:37,341::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70970::DEBUG::2014-09-25 17:43:37,341::task::993::Storage.TaskManager.Task::(_decref) Task=`1bef0965-48d2-45ec-8d3e-2bd78053a02b`::ref 0 aborting False >Thread-70970::DEBUG::2014-09-25 17:43:37,343::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:37,365::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:37,366::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70971::DEBUG::2014-09-25 17:43:37,368::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:37,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:37,743::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0217952 s, 47.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:38,051::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:38,053::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70972::DEBUG::2014-09-25 17:43:38,053::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70972::DEBUG::2014-09-25 17:43:38,055::task::595::Storage.TaskManager.Task::(_updateState) Task=`375a243d-df8d-41a1-bd7b-a40a0afefa8b`::moving from state init -> state preparing >Thread-70972::INFO::2014-09-25 17:43:38,055::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70972::INFO::2014-09-25 17:43:38,056::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70972::DEBUG::2014-09-25 17:43:38,056::task::1191::Storage.TaskManager.Task::(prepare) Task=`375a243d-df8d-41a1-bd7b-a40a0afefa8b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70972::DEBUG::2014-09-25 17:43:38,056::task::595::Storage.TaskManager.Task::(_updateState) Task=`375a243d-df8d-41a1-bd7b-a40a0afefa8b`::moving from state preparing -> state finished >Thread-70972::DEBUG::2014-09-25 17:43:38,056::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70972::DEBUG::2014-09-25 17:43:38,057::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70972::DEBUG::2014-09-25 17:43:38,057::task::993::Storage.TaskManager.Task::(_decref) Task=`375a243d-df8d-41a1-bd7b-a40a0afefa8b`::ref 0 aborting False >Thread-70972::DEBUG::2014-09-25 17:43:38,057::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70972::DEBUG::2014-09-25 17:43:38,058::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:38,070::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70973::DEBUG::2014-09-25 17:43:38,071::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:43:38,071::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70973::DEBUG::2014-09-25 17:43:38,073::task::595::Storage.TaskManager.Task::(_updateState) Task=`e6a1896e-9dbb-4cef-b2d7-15e60c977b6b`::moving from state init -> state preparing >Thread-70973::INFO::2014-09-25 17:43:38,074::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70973::DEBUG::2014-09-25 17:43:38,074::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f798ad83-aab2-4641-9246-9a771b3a68ec`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70973::DEBUG::2014-09-25 17:43:38,074::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70973::DEBUG::2014-09-25 17:43:38,075::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70973::DEBUG::2014-09-25 17:43:38,075::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f798ad83-aab2-4641-9246-9a771b3a68ec`::Granted request >Thread-70973::DEBUG::2014-09-25 17:43:38,077::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e6a1896e-9dbb-4cef-b2d7-15e60c977b6b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70973::DEBUG::2014-09-25 17:43:38,077::task::993::Storage.TaskManager.Task::(_decref) Task=`e6a1896e-9dbb-4cef-b2d7-15e60c977b6b`::ref 1 aborting False >Thread-70973::INFO::2014-09-25 17:43:38,081::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70973::DEBUG::2014-09-25 17:43:38,082::task::1191::Storage.TaskManager.Task::(prepare) Task=`e6a1896e-9dbb-4cef-b2d7-15e60c977b6b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70973::DEBUG::2014-09-25 17:43:38,082::task::595::Storage.TaskManager.Task::(_updateState) Task=`e6a1896e-9dbb-4cef-b2d7-15e60c977b6b`::moving from state preparing -> state finished >Thread-70973::DEBUG::2014-09-25 17:43:38,082::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70973::DEBUG::2014-09-25 17:43:38,082::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70973::DEBUG::2014-09-25 17:43:38,083::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70973::DEBUG::2014-09-25 17:43:38,083::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70973::DEBUG::2014-09-25 17:43:38,083::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70973::DEBUG::2014-09-25 17:43:38,084::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70973::DEBUG::2014-09-25 17:43:38,084::task::993::Storage.TaskManager.Task::(_decref) Task=`e6a1896e-9dbb-4cef-b2d7-15e60c977b6b`::ref 0 aborting False >Thread-70973::DEBUG::2014-09-25 17:43:38,084::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70973::DEBUG::2014-09-25 17:43:38,085::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:43:38,786::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:38,786::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:38,786::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:38,787::__init__::232::IOProcess::(_processLogs) (109187) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:38,788::__init__::232::IOProcess::(_processLogs) (109187) Queuing response >Thread-79::DEBUG::2014-09-25 17:43:38,789::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:43:38,802::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00361968 s, 155 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:38,804::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:38,804::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:38,804::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:38,804::__init__::232::IOProcess::(_processLogs) (109188) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:38,806::__init__::232::IOProcess::(_processLogs) (109188) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:38,807::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:38,808::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:38,808::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:38,808::__init__::232::IOProcess::(_processLogs) (109189) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:38,809::__init__::232::IOProcess::(_processLogs) (109189) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:38,809::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:38,809::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:38,810::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:38,810::__init__::232::IOProcess::(_processLogs) (109190) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:38,810::__init__::232::IOProcess::(_processLogs) (109190) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:38,811::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:38,812::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:38,813::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:38,813::__init__::232::IOProcess::(_processLogs) (109191) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:38,813::__init__::232::IOProcess::(_processLogs) (109191) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:39,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:39,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132561 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:40,427::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:40,428::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70974::DEBUG::2014-09-25 17:43:40,430::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:41,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:41,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163787 s, 62.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:43,418::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:43,419::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:43,419::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:43,419::__init__::232::IOProcess::(_processLogs) (109192) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:43,424::__init__::232::IOProcess::(_processLogs) (109192) Queuing response >Thread-81::DEBUG::2014-09-25 17:43:43,426::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:43:43,444::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00592607 s, 62.6 kB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:43,446::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-29::DEBUG::2014-09-25 17:43:43,447::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:43,449::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >JsonRpcServer::DEBUG::2014-09-25 17:43:43,450::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-29::DEBUG::2014-09-25 17:43:43,451::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:43,453::__init__::232::IOProcess::(_processLogs) (109193) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:43,455::__init__::232::IOProcess::(_processLogs) (109193) Queuing response >Thread-70975::DEBUG::2014-09-25 17:43:43,457::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:43,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:43,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186508 s, 54.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:45,829::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:45,830::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:45,830::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:45,830::__init__::232::IOProcess::(_processLogs) (109194) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:45,831::__init__::232::IOProcess::(_processLogs) (109194) Queuing response >Thread-80::DEBUG::2014-09-25 17:43:45,832::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:43:45,841::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000372678 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:45,843::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:45,843::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:45,843::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:45,843::__init__::232::IOProcess::(_processLogs) (109195) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:45,844::__init__::232::IOProcess::(_processLogs) (109195) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:45,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:45,923::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132676 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:46,473::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:46,477::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70976::DEBUG::2014-09-25 17:43:46,482::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:47,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:47,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169967 s, 60.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:48,112::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:48,114::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70977::DEBUG::2014-09-25 17:43:48,114::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70977::DEBUG::2014-09-25 17:43:48,116::task::595::Storage.TaskManager.Task::(_updateState) Task=`f0d34d54-a515-4395-9929-4d30f0347ce3`::moving from state init -> state preparing >Thread-70977::INFO::2014-09-25 17:43:48,116::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70977::INFO::2014-09-25 17:43:48,117::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70977::DEBUG::2014-09-25 17:43:48,117::task::1191::Storage.TaskManager.Task::(prepare) Task=`f0d34d54-a515-4395-9929-4d30f0347ce3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70977::DEBUG::2014-09-25 17:43:48,117::task::595::Storage.TaskManager.Task::(_updateState) Task=`f0d34d54-a515-4395-9929-4d30f0347ce3`::moving from state preparing -> state finished >Thread-70977::DEBUG::2014-09-25 17:43:48,117::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70977::DEBUG::2014-09-25 17:43:48,118::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70977::DEBUG::2014-09-25 17:43:48,118::task::993::Storage.TaskManager.Task::(_decref) Task=`f0d34d54-a515-4395-9929-4d30f0347ce3`::ref 0 aborting False >Thread-70977::DEBUG::2014-09-25 17:43:48,118::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70977::DEBUG::2014-09-25 17:43:48,119::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:48,132::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:48,133::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70978::DEBUG::2014-09-25 17:43:48,133::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70978::DEBUG::2014-09-25 17:43:48,135::task::595::Storage.TaskManager.Task::(_updateState) Task=`301a23d1-8016-47f4-97da-aa08a1008d27`::moving from state init -> state preparing >Thread-70978::INFO::2014-09-25 17:43:48,135::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70978::DEBUG::2014-09-25 17:43:48,136::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7ee3ddc5-d5bb-4014-848a-4c63f7998c09`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70978::DEBUG::2014-09-25 17:43:48,136::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70978::DEBUG::2014-09-25 17:43:48,136::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70978::DEBUG::2014-09-25 17:43:48,137::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7ee3ddc5-d5bb-4014-848a-4c63f7998c09`::Granted request >Thread-70978::DEBUG::2014-09-25 17:43:48,137::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`301a23d1-8016-47f4-97da-aa08a1008d27`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70978::DEBUG::2014-09-25 17:43:48,137::task::993::Storage.TaskManager.Task::(_decref) Task=`301a23d1-8016-47f4-97da-aa08a1008d27`::ref 1 aborting False >Thread-70978::INFO::2014-09-25 17:43:48,142::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70978::DEBUG::2014-09-25 17:43:48,142::task::1191::Storage.TaskManager.Task::(prepare) Task=`301a23d1-8016-47f4-97da-aa08a1008d27`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70978::DEBUG::2014-09-25 17:43:48,142::task::595::Storage.TaskManager.Task::(_updateState) Task=`301a23d1-8016-47f4-97da-aa08a1008d27`::moving from state preparing -> state finished >Thread-70978::DEBUG::2014-09-25 17:43:48,143::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70978::DEBUG::2014-09-25 17:43:48,143::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70978::DEBUG::2014-09-25 17:43:48,143::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70978::DEBUG::2014-09-25 17:43:48,143::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70978::DEBUG::2014-09-25 17:43:48,144::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70978::DEBUG::2014-09-25 17:43:48,144::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70978::DEBUG::2014-09-25 17:43:48,144::task::993::Storage.TaskManager.Task::(_decref) Task=`301a23d1-8016-47f4-97da-aa08a1008d27`::ref 0 aborting False >Thread-70978::DEBUG::2014-09-25 17:43:48,145::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70978::DEBUG::2014-09-25 17:43:48,145::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-79::DEBUG::2014-09-25 17:43:48,813::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:43:48,817::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,817::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,817::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,818::__init__::232::IOProcess::(_processLogs) (109196) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:43:48,818::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,818::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,819::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,819::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,819::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,819::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,820::__init__::232::IOProcess::(_processLogs) (109197) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:43:48,820::__init__::232::IOProcess::(_processLogs) (109198) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:43:48,820::__init__::232::IOProcess::(_processLogs) (109197) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:48,821::__init__::232::IOProcess::(_processLogs) (109196) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:48,825::__init__::232::IOProcess::(_processLogs) (109198) Queuing response >Thread-79::DEBUG::2014-09-25 17:43:48,829::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:43:48,830::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,830::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,830::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,831::__init__::232::IOProcess::(_processLogs) (109199) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:43:48,895::__init__::232::IOProcess::(_processLogs) (109199) Queuing response >Thread-79::DEBUG::2014-09-25 17:43:48,896::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:43:48,897::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,897::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,898::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,898::__init__::232::IOProcess::(_processLogs) (109200) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:43:48,912::__init__::232::IOProcess::(_processLogs) (109200) Queuing response >Thread-79::DEBUG::2014-09-25 17:43:48,912::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:43:48,914::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,914::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,914::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,915::__init__::232::IOProcess::(_processLogs) (109201) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:48,915::__init__::232::IOProcess::(_processLogs) (109201) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:48,916::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,916::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,916::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,916::__init__::232::IOProcess::(_processLogs) (109202) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:43:48,925::__init__::232::IOProcess::(_processLogs) (109202) Queuing response >Thread-79::DEBUG::2014-09-25 17:43:48,925::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 17:43:48,925::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:43:48,926::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:43:48,926::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,927::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,927::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,927::__init__::232::IOProcess::(_processLogs) (109203) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:48,930::__init__::232::IOProcess::(_processLogs) (109203) Queuing response >Thread-79::DEBUG::2014-09-25 17:43:48,931::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:43:48,947::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00604121 s, 93.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:48,948::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,948::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,948::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,949::__init__::232::IOProcess::(_processLogs) (109204) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:48,955::__init__::232::IOProcess::(_processLogs) (109204) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:48,956::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,957::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,957::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,957::__init__::232::IOProcess::(_processLogs) (109205) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:48,964::__init__::232::IOProcess::(_processLogs) (109205) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:48,965::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,965::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,965::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,966::__init__::232::IOProcess::(_processLogs) (109206) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:48,975::__init__::232::IOProcess::(_processLogs) (109206) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:48,976::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:48,976::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:48,976::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:48,976::__init__::232::IOProcess::(_processLogs) (109207) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:48,981::__init__::232::IOProcess::(_processLogs) (109207) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:49,499::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:49,500::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70982::DEBUG::2014-09-25 17:43:49,502::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:49,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:50,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0255974 s, 40.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:43:52,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:52,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131296 s, 78.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:52,518::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:52,520::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70983::DEBUG::2014-09-25 17:43:52,522::task::595::Storage.TaskManager.Task::(_updateState) Task=`b3faa8a3-8e7c-4802-bc88-3b0d94dcc15a`::moving from state init -> state preparing >Thread-70983::INFO::2014-09-25 17:43:52,523::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70983::INFO::2014-09-25 17:43:52,523::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00604121', 'lastCheck': '3.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000372678', 'lastCheck': '6.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00592607', 'lastCheck': '9.1', 'valid': True}} >Thread-70983::DEBUG::2014-09-25 17:43:52,524::task::1191::Storage.TaskManager.Task::(prepare) Task=`b3faa8a3-8e7c-4802-bc88-3b0d94dcc15a`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00604121', 'lastCheck': '3.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000372678', 'lastCheck': '6.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00592607', 'lastCheck': '9.1', 'valid': True}} >Thread-70983::DEBUG::2014-09-25 17:43:52,524::task::595::Storage.TaskManager.Task::(_updateState) Task=`b3faa8a3-8e7c-4802-bc88-3b0d94dcc15a`::moving from state preparing -> state finished >Thread-70983::DEBUG::2014-09-25 17:43:52,524::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70983::DEBUG::2014-09-25 17:43:52,524::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70983::DEBUG::2014-09-25 17:43:52,524::task::993::Storage.TaskManager.Task::(_decref) Task=`b3faa8a3-8e7c-4802-bc88-3b0d94dcc15a`::ref 0 aborting False >Thread-70983::DEBUG::2014-09-25 17:43:52,527::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:52,549::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:52,550::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70984::DEBUG::2014-09-25 17:43:52,552::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:43:53,458::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:53,458::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:53,458::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:53,458::__init__::232::IOProcess::(_processLogs) (109208) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:53,459::__init__::232::IOProcess::(_processLogs) (109208) Queuing response >Thread-81::DEBUG::2014-09-25 17:43:53,461::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:43:53,478::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00583202 s, 63.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:53,479::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:53,480::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:53,480::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:53,480::__init__::232::IOProcess::(_processLogs) (109209) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:53,484::__init__::232::IOProcess::(_processLogs) (109209) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:54,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:54,118::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014148 s, 72.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:55,614::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:55,615::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70985::DEBUG::2014-09-25 17:43:55,617::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:43:55,846::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:55,847::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:55,847::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:55,847::__init__::232::IOProcess::(_processLogs) (109210) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:55,848::__init__::232::IOProcess::(_processLogs) (109210) Queuing response >Thread-80::DEBUG::2014-09-25 17:43:55,849::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:43:55,858::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000486948 s, 776 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:55,859::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:55,859::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:55,860::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:55,860::__init__::232::IOProcess::(_processLogs) (109211) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:55,860::__init__::232::IOProcess::(_processLogs) (109211) Queuing response >Dummy-148::DEBUG::2014-09-25 17:43:56,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:43:56,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211802 s, 48.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:58,174::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70986::DEBUG::2014-09-25 17:43:58,175::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:43:58,175::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70986::DEBUG::2014-09-25 17:43:58,177::task::595::Storage.TaskManager.Task::(_updateState) Task=`e9812597-4942-41f4-a09e-1c3a24dbae48`::moving from state init -> state preparing >Thread-70986::INFO::2014-09-25 17:43:58,178::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70986::INFO::2014-09-25 17:43:58,178::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70986::DEBUG::2014-09-25 17:43:58,179::task::1191::Storage.TaskManager.Task::(prepare) Task=`e9812597-4942-41f4-a09e-1c3a24dbae48`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70986::DEBUG::2014-09-25 17:43:58,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`e9812597-4942-41f4-a09e-1c3a24dbae48`::moving from state preparing -> state finished >Thread-70986::DEBUG::2014-09-25 17:43:58,179::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70986::DEBUG::2014-09-25 17:43:58,179::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70986::DEBUG::2014-09-25 17:43:58,179::task::993::Storage.TaskManager.Task::(_decref) Task=`e9812597-4942-41f4-a09e-1c3a24dbae48`::ref 0 aborting False >Thread-70986::DEBUG::2014-09-25 17:43:58,180::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70986::DEBUG::2014-09-25 17:43:58,180::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:58,191::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:58,199::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:58,201::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70987::DEBUG::2014-09-25 17:43:58,201::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-70987::DEBUG::2014-09-25 17:43:58,203::task::595::Storage.TaskManager.Task::(_updateState) Task=`a47f357c-6c58-469f-ab02-16dd2e128c5b`::moving from state init -> state preparing >Thread-70987::INFO::2014-09-25 17:43:58,203::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70987::DEBUG::2014-09-25 17:43:58,204::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0d6fa432-a66c-4e7b-a393-da76edfd8cd1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70987::DEBUG::2014-09-25 17:43:58,204::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70987::DEBUG::2014-09-25 17:43:58,205::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70987::DEBUG::2014-09-25 17:43:58,205::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0d6fa432-a66c-4e7b-a393-da76edfd8cd1`::Granted request >Thread-70987::DEBUG::2014-09-25 17:43:58,206::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a47f357c-6c58-469f-ab02-16dd2e128c5b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70987::DEBUG::2014-09-25 17:43:58,206::task::993::Storage.TaskManager.Task::(_decref) Task=`a47f357c-6c58-469f-ab02-16dd2e128c5b`::ref 1 aborting False >Thread-70987::INFO::2014-09-25 17:43:58,211::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70987::DEBUG::2014-09-25 17:43:58,211::task::1191::Storage.TaskManager.Task::(prepare) Task=`a47f357c-6c58-469f-ab02-16dd2e128c5b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70987::DEBUG::2014-09-25 17:43:58,211::task::595::Storage.TaskManager.Task::(_updateState) Task=`a47f357c-6c58-469f-ab02-16dd2e128c5b`::moving from state preparing -> state finished >Thread-70987::DEBUG::2014-09-25 17:43:58,212::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70987::DEBUG::2014-09-25 17:43:58,212::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70987::DEBUG::2014-09-25 17:43:58,212::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70987::DEBUG::2014-09-25 17:43:58,213::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70987::DEBUG::2014-09-25 17:43:58,213::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70987::DEBUG::2014-09-25 17:43:58,213::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70987::DEBUG::2014-09-25 17:43:58,213::task::993::Storage.TaskManager.Task::(_decref) Task=`a47f357c-6c58-469f-ab02-16dd2e128c5b`::ref 0 aborting False >Thread-70987::DEBUG::2014-09-25 17:43:58,214::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70987::DEBUG::2014-09-25 17:43:58,214::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:43:58,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0234006 s, 43.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:43:58,632::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:43:58,634::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70988::DEBUG::2014-09-25 17:43:58,636::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:43:58,983::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:58,983::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:58,983::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:58,984::__init__::232::IOProcess::(_processLogs) (109212) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:58,988::__init__::232::IOProcess::(_processLogs) (109212) Queuing response >Thread-79::DEBUG::2014-09-25 17:43:58,989::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:43:59,014::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00697213 s, 80.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:43:59,015::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:59,015::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:59,016::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:59,016::__init__::232::IOProcess::(_processLogs) (109213) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:43:59,022::__init__::232::IOProcess::(_processLogs) (109213) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:59,023::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:59,023::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:59,024::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:59,024::__init__::232::IOProcess::(_processLogs) (109214) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:59,024::__init__::232::IOProcess::(_processLogs) (109214) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:59,025::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:59,025::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:59,026::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:59,026::__init__::232::IOProcess::(_processLogs) (109215) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:59,027::__init__::232::IOProcess::(_processLogs) (109215) Queuing response >Thread-29::DEBUG::2014-09-25 17:43:59,027::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:43:59,028::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:43:59,028::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:43:59,029::__init__::232::IOProcess::(_processLogs) (109216) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:43:59,029::__init__::232::IOProcess::(_processLogs) (109216) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:00,251::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:00,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135718 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:01,650::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:01,651::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70989::DEBUG::2014-09-25 17:44:01,653::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:02,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:02,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0273942 s, 37.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:03,486::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:03,486::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:03,487::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:03,487::__init__::232::IOProcess::(_processLogs) (109217) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:03,495::__init__::232::IOProcess::(_processLogs) (109217) Queuing response >Thread-81::DEBUG::2014-09-25 17:44:03,496::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:44:03,516::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00290943 s, 128 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:03,517::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:03,517::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:03,517::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:03,518::__init__::232::IOProcess::(_processLogs) (109218) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:03,522::__init__::232::IOProcess::(_processLogs) (109218) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:04,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:04,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151716 s, 67.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:04,670::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:04,671::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70990::DEBUG::2014-09-25 17:44:04,673::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:44:05,862::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:05,863::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:05,863::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:05,863::__init__::232::IOProcess::(_processLogs) (109219) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:05,864::__init__::232::IOProcess::(_processLogs) (109219) Queuing response >Thread-80::DEBUG::2014-09-25 17:44:05,865::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:44:05,874::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000389329 s, 971 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:05,876::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:05,878::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:05,878::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:05,878::__init__::232::IOProcess::(_processLogs) (109220) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:05,879::__init__::232::IOProcess::(_processLogs) (109220) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:06,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:06,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135414 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:07,689::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:07,690::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70991::DEBUG::2014-09-25 17:44:07,693::task::595::Storage.TaskManager.Task::(_updateState) Task=`7497fa16-18c7-4de4-aa1b-31ad70443f72`::moving from state init -> state preparing >Thread-70991::INFO::2014-09-25 17:44:07,693::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-70991::INFO::2014-09-25 17:44:07,694::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00697213', 'lastCheck': '8.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000389329', 'lastCheck': '1.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00290943', 'lastCheck': '4.2', 'valid': True}} >Thread-70991::DEBUG::2014-09-25 17:44:07,694::task::1191::Storage.TaskManager.Task::(prepare) Task=`7497fa16-18c7-4de4-aa1b-31ad70443f72`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00697213', 'lastCheck': '8.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000389329', 'lastCheck': '1.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00290943', 'lastCheck': '4.2', 'valid': True}} >Thread-70991::DEBUG::2014-09-25 17:44:07,694::task::595::Storage.TaskManager.Task::(_updateState) Task=`7497fa16-18c7-4de4-aa1b-31ad70443f72`::moving from state preparing -> state finished >Thread-70991::DEBUG::2014-09-25 17:44:07,694::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70991::DEBUG::2014-09-25 17:44:07,695::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70991::DEBUG::2014-09-25 17:44:07,695::task::993::Storage.TaskManager.Task::(_decref) Task=`7497fa16-18c7-4de4-aa1b-31ad70443f72`::ref 0 aborting False >Thread-70991::DEBUG::2014-09-25 17:44:07,697::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:07,717::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:07,718::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70992::DEBUG::2014-09-25 17:44:07,720::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:08,245::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70993::DEBUG::2014-09-25 17:44:08,246::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:44:08,246::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70993::DEBUG::2014-09-25 17:44:08,248::task::595::Storage.TaskManager.Task::(_updateState) Task=`7a302c21-ace2-4bde-a395-a696f703fb3b`::moving from state init -> state preparing >Thread-70993::INFO::2014-09-25 17:44:08,249::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70993::INFO::2014-09-25 17:44:08,249::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70993::DEBUG::2014-09-25 17:44:08,249::task::1191::Storage.TaskManager.Task::(prepare) Task=`7a302c21-ace2-4bde-a395-a696f703fb3b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-70993::DEBUG::2014-09-25 17:44:08,250::task::595::Storage.TaskManager.Task::(_updateState) Task=`7a302c21-ace2-4bde-a395-a696f703fb3b`::moving from state preparing -> state finished >Thread-70993::DEBUG::2014-09-25 17:44:08,250::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-70993::DEBUG::2014-09-25 17:44:08,250::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70993::DEBUG::2014-09-25 17:44:08,250::task::993::Storage.TaskManager.Task::(_decref) Task=`7a302c21-ace2-4bde-a395-a696f703fb3b`::ref 0 aborting False >Thread-70993::DEBUG::2014-09-25 17:44:08,251::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-70993::DEBUG::2014-09-25 17:44:08,251::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:08,265::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-70994::DEBUG::2014-09-25 17:44:08,266::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:44:08,266::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70994::DEBUG::2014-09-25 17:44:08,268::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f0c35f6-3e13-44c1-a1da-c7f1070f03d0`::moving from state init -> state preparing >Thread-70994::INFO::2014-09-25 17:44:08,268::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-70994::DEBUG::2014-09-25 17:44:08,269::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0ec309c3-a13a-40c9-9775-f7147dca25ba`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-70994::DEBUG::2014-09-25 17:44:08,269::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-70994::DEBUG::2014-09-25 17:44:08,269::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-70994::DEBUG::2014-09-25 17:44:08,270::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0ec309c3-a13a-40c9-9775-f7147dca25ba`::Granted request >Thread-70994::DEBUG::2014-09-25 17:44:08,270::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4f0c35f6-3e13-44c1-a1da-c7f1070f03d0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-70994::DEBUG::2014-09-25 17:44:08,271::task::993::Storage.TaskManager.Task::(_decref) Task=`4f0c35f6-3e13-44c1-a1da-c7f1070f03d0`::ref 1 aborting False >Thread-70994::INFO::2014-09-25 17:44:08,275::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70994::DEBUG::2014-09-25 17:44:08,275::task::1191::Storage.TaskManager.Task::(prepare) Task=`4f0c35f6-3e13-44c1-a1da-c7f1070f03d0`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70994::DEBUG::2014-09-25 17:44:08,275::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f0c35f6-3e13-44c1-a1da-c7f1070f03d0`::moving from state preparing -> state finished >Thread-70994::DEBUG::2014-09-25 17:44:08,276::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-70994::DEBUG::2014-09-25 17:44:08,276::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-70994::DEBUG::2014-09-25 17:44:08,276::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-70994::DEBUG::2014-09-25 17:44:08,276::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-70994::DEBUG::2014-09-25 17:44:08,277::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-70994::DEBUG::2014-09-25 17:44:08,277::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-70994::DEBUG::2014-09-25 17:44:08,277::task::993::Storage.TaskManager.Task::(_decref) Task=`4f0c35f6-3e13-44c1-a1da-c7f1070f03d0`::ref 0 aborting False >Thread-70994::DEBUG::2014-09-25 17:44:08,278::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-70994::DEBUG::2014-09-25 17:44:08,278::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:08,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:08,470::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171145 s, 59.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:09,029::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:09,030::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:09,030::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:09,030::__init__::232::IOProcess::(_processLogs) (109221) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:09,037::__init__::232::IOProcess::(_processLogs) (109221) Queuing response >Thread-79::DEBUG::2014-09-25 17:44:09,039::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:44:09,064::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00295867 s, 190 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:09,065::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:09,066::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:09,066::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:09,066::__init__::232::IOProcess::(_processLogs) (109222) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:09,072::__init__::232::IOProcess::(_processLogs) (109222) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:09,073::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:09,074::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:09,074::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:09,075::__init__::232::IOProcess::(_processLogs) (109223) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:09,075::__init__::232::IOProcess::(_processLogs) (109223) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:09,075::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:09,076::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:09,076::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:09,077::__init__::232::IOProcess::(_processLogs) (109224) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:09,077::__init__::232::IOProcess::(_processLogs) (109224) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:09,078::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:09,078::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:09,078::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:09,079::__init__::232::IOProcess::(_processLogs) (109225) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:09,079::__init__::232::IOProcess::(_processLogs) (109225) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:10,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:10,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175985 s, 58.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:10,778::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:10,780::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70995::DEBUG::2014-09-25 17:44:10,782::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:12,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:12,581::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210144 s, 48.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:13,524::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:13,525::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:13,525::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:13,525::__init__::232::IOProcess::(_processLogs) (109226) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:13,527::__init__::232::IOProcess::(_processLogs) (109226) Queuing response >Thread-81::DEBUG::2014-09-25 17:44:13,528::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:44:13,543::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00381523 s, 97.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:13,545::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:13,545::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:13,545::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:13,545::__init__::232::IOProcess::(_processLogs) (109227) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:13,548::__init__::232::IOProcess::(_processLogs) (109227) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:13,798::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:13,800::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-70996::DEBUG::2014-09-25 17:44:13,802::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:14,602::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:14,635::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185446 s, 55.2 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 17:44:15,880::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:44:15,885::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:15,885::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:15,886::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:15,886::__init__::232::IOProcess::(_processLogs) (109228) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:44:15,886::__init__::232::IOProcess::(_processLogs) (109228) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:15,887::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:15,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:15,888::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:15,888::__init__::232::IOProcess::(_processLogs) (109229) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:44:15,889::__init__::232::IOProcess::(_processLogs) (109229) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:15,891::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:15,892::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:15,895::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:15,897::__init__::232::IOProcess::(_processLogs) (109230) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:44:15,899::__init__::232::IOProcess::(_processLogs) (109230) Queuing response >Thread-80::DEBUG::2014-09-25 17:44:15,901::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:44:15,902::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:15,902::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:15,903::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:15,903::__init__::232::IOProcess::(_processLogs) (109231) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:44:15,906::__init__::232::IOProcess::(_processLogs) (109231) Queuing response >Thread-80::DEBUG::2014-09-25 17:44:15,906::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:44:15,908::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:15,908::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:15,908::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:15,909::__init__::232::IOProcess::(_processLogs) (109232) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:44:15,909::__init__::232::IOProcess::(_processLogs) (109232) Queuing response >Thread-80::DEBUG::2014-09-25 17:44:15,910::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:44:15,911::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:15,912::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:15,912::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:15,913::__init__::232::IOProcess::(_processLogs) (109233) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:15,913::__init__::232::IOProcess::(_processLogs) (109233) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:15,913::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:15,914::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:15,914::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:15,914::__init__::232::IOProcess::(_processLogs) (109234) Got request for method 'glob' >Thread-80::DEBUG::2014-09-25 17:44:15,915::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:44:15,915::__init__::232::IOProcess::(_processLogs) (109234) Queuing response >Thread-80::INFO::2014-09-25 17:44:15,916::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:44:15,916::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:44:15,917::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:15,917::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:15,918::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:44:15,919::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:44:15,919::__init__::232::IOProcess::(_processLogs) (109235) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:15,928::__init__::232::IOProcess::(_processLogs) (109235) Queuing response >Thread-80::DEBUG::2014-09-25 17:44:15,929::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000439567 s, 860 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:15,930::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:15,930::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:15,930::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:15,931::__init__::232::IOProcess::(_processLogs) (109236) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:15,931::__init__::232::IOProcess::(_processLogs) (109236) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:16,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:16,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0263566 s, 38.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:16,821::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:16,825::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71000::DEBUG::2014-09-25 17:44:16,833::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:18,305::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71001::DEBUG::2014-09-25 17:44:18,306::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:44:18,306::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71001::DEBUG::2014-09-25 17:44:18,308::task::595::Storage.TaskManager.Task::(_updateState) Task=`1438a9d0-721a-424b-8d92-6525667fbdec`::moving from state init -> state preparing >Thread-71001::INFO::2014-09-25 17:44:18,309::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71001::INFO::2014-09-25 17:44:18,309::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71001::DEBUG::2014-09-25 17:44:18,309::task::1191::Storage.TaskManager.Task::(prepare) Task=`1438a9d0-721a-424b-8d92-6525667fbdec`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71001::DEBUG::2014-09-25 17:44:18,310::task::595::Storage.TaskManager.Task::(_updateState) Task=`1438a9d0-721a-424b-8d92-6525667fbdec`::moving from state preparing -> state finished >Thread-71001::DEBUG::2014-09-25 17:44:18,310::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71001::DEBUG::2014-09-25 17:44:18,310::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71001::DEBUG::2014-09-25 17:44:18,310::task::993::Storage.TaskManager.Task::(_decref) Task=`1438a9d0-721a-424b-8d92-6525667fbdec`::ref 0 aborting False >Thread-71001::DEBUG::2014-09-25 17:44:18,311::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71001::DEBUG::2014-09-25 17:44:18,311::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:18,325::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:18,326::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71002::DEBUG::2014-09-25 17:44:18,326::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71002::DEBUG::2014-09-25 17:44:18,328::task::595::Storage.TaskManager.Task::(_updateState) Task=`09d3603c-ad33-44cc-a635-e78b378bae8f`::moving from state init -> state preparing >Thread-71002::INFO::2014-09-25 17:44:18,328::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71002::DEBUG::2014-09-25 17:44:18,329::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4678485f-2a76-4d4f-9269-8c5d7e349fdb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71002::DEBUG::2014-09-25 17:44:18,329::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71002::DEBUG::2014-09-25 17:44:18,330::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71002::DEBUG::2014-09-25 17:44:18,330::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4678485f-2a76-4d4f-9269-8c5d7e349fdb`::Granted request >Thread-71002::DEBUG::2014-09-25 17:44:18,332::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`09d3603c-ad33-44cc-a635-e78b378bae8f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71002::DEBUG::2014-09-25 17:44:18,332::task::993::Storage.TaskManager.Task::(_decref) Task=`09d3603c-ad33-44cc-a635-e78b378bae8f`::ref 1 aborting False >Thread-71002::INFO::2014-09-25 17:44:18,336::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71002::DEBUG::2014-09-25 17:44:18,336::task::1191::Storage.TaskManager.Task::(prepare) Task=`09d3603c-ad33-44cc-a635-e78b378bae8f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71002::DEBUG::2014-09-25 17:44:18,337::task::595::Storage.TaskManager.Task::(_updateState) Task=`09d3603c-ad33-44cc-a635-e78b378bae8f`::moving from state preparing -> state finished >Thread-71002::DEBUG::2014-09-25 17:44:18,337::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71002::DEBUG::2014-09-25 17:44:18,337::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71002::DEBUG::2014-09-25 17:44:18,338::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71002::DEBUG::2014-09-25 17:44:18,338::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71002::DEBUG::2014-09-25 17:44:18,338::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71002::DEBUG::2014-09-25 17:44:18,338::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71002::DEBUG::2014-09-25 17:44:18,339::task::993::Storage.TaskManager.Task::(_decref) Task=`09d3603c-ad33-44cc-a635-e78b378bae8f`::ref 0 aborting False >Thread-71002::DEBUG::2014-09-25 17:44:18,339::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71002::DEBUG::2014-09-25 17:44:18,340::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:18,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:18,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0300601 s, 34.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:19,080::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:19,080::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:19,080::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:19,081::__init__::232::IOProcess::(_processLogs) (109237) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:19,082::__init__::232::IOProcess::(_processLogs) (109237) Queuing response >Thread-79::DEBUG::2014-09-25 17:44:19,084::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:44:19,097::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00277467 s, 203 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:19,098::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:19,098::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:19,099::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:19,099::__init__::232::IOProcess::(_processLogs) (109238) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:19,101::__init__::232::IOProcess::(_processLogs) (109238) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:19,103::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:19,104::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:19,104::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:19,105::__init__::232::IOProcess::(_processLogs) (109239) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:19,105::__init__::232::IOProcess::(_processLogs) (109239) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:19,105::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:19,106::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:19,106::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:19,107::__init__::232::IOProcess::(_processLogs) (109240) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:19,107::__init__::232::IOProcess::(_processLogs) (109240) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:19,107::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:19,108::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:19,109::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:19,109::__init__::232::IOProcess::(_processLogs) (109241) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:19,109::__init__::232::IOProcess::(_processLogs) (109241) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:19,853::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:19,854::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71003::DEBUG::2014-09-25 17:44:19,856::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:20,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:21,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131647 s, 77.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:22,870::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:22,871::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71004::DEBUG::2014-09-25 17:44:22,874::task::595::Storage.TaskManager.Task::(_updateState) Task=`df7acbe3-b21d-40ec-bd01-5bd6be9ef91a`::moving from state init -> state preparing >Thread-71004::INFO::2014-09-25 17:44:22,874::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71004::INFO::2014-09-25 17:44:22,874::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00277467', 'lastCheck': '3.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000439567', 'lastCheck': '6.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00381523', 'lastCheck': '9.3', 'valid': True}} >Thread-71004::DEBUG::2014-09-25 17:44:22,875::task::1191::Storage.TaskManager.Task::(prepare) Task=`df7acbe3-b21d-40ec-bd01-5bd6be9ef91a`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00277467', 'lastCheck': '3.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000439567', 'lastCheck': '6.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00381523', 'lastCheck': '9.3', 'valid': True}} >Thread-71004::DEBUG::2014-09-25 17:44:22,875::task::595::Storage.TaskManager.Task::(_updateState) Task=`df7acbe3-b21d-40ec-bd01-5bd6be9ef91a`::moving from state preparing -> state finished >Thread-71004::DEBUG::2014-09-25 17:44:22,875::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71004::DEBUG::2014-09-25 17:44:22,875::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71004::DEBUG::2014-09-25 17:44:22,876::task::993::Storage.TaskManager.Task::(_decref) Task=`df7acbe3-b21d-40ec-bd01-5bd6be9ef91a`::ref 0 aborting False >Thread-71004::DEBUG::2014-09-25 17:44:22,878::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:22,897::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:22,898::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71005::DEBUG::2014-09-25 17:44:22,900::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:23,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:23,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013024 s, 78.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:23,550::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:23,550::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:23,550::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:23,551::__init__::232::IOProcess::(_processLogs) (109242) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:23,552::__init__::232::IOProcess::(_processLogs) (109242) Queuing response >Thread-81::DEBUG::2014-09-25 17:44:23,553::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:44:23,569::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00355564 s, 104 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:23,571::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:23,571::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:23,571::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:23,571::__init__::232::IOProcess::(_processLogs) (109243) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:23,572::__init__::232::IOProcess::(_processLogs) (109243) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:25,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:25,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164384 s, 62.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:25,934::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:25,934::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:25,934::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:25,935::__init__::232::IOProcess::(_processLogs) (109244) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:25,935::__init__::232::IOProcess::(_processLogs) (109244) Queuing response >Thread-80::DEBUG::2014-09-25 17:44:25,935::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:44:25,944::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000494592 s, 764 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:25,945::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:25,946::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:25,946::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:25,946::__init__::232::IOProcess::(_processLogs) (109245) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:25,947::__init__::232::IOProcess::(_processLogs) (109245) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:25,955::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:25,957::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71006::DEBUG::2014-09-25 17:44:25,959::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:27,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:27,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132756 s, 77.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:28,370::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:28,372::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71007::DEBUG::2014-09-25 17:44:28,372::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71007::DEBUG::2014-09-25 17:44:28,374::task::595::Storage.TaskManager.Task::(_updateState) Task=`b4a61679-e63d-45bc-8d8c-53031fdbca31`::moving from state init -> state preparing >Thread-71007::INFO::2014-09-25 17:44:28,374::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71007::INFO::2014-09-25 17:44:28,375::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71007::DEBUG::2014-09-25 17:44:28,375::task::1191::Storage.TaskManager.Task::(prepare) Task=`b4a61679-e63d-45bc-8d8c-53031fdbca31`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71007::DEBUG::2014-09-25 17:44:28,375::task::595::Storage.TaskManager.Task::(_updateState) Task=`b4a61679-e63d-45bc-8d8c-53031fdbca31`::moving from state preparing -> state finished >Thread-71007::DEBUG::2014-09-25 17:44:28,375::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71007::DEBUG::2014-09-25 17:44:28,376::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71007::DEBUG::2014-09-25 17:44:28,376::task::993::Storage.TaskManager.Task::(_decref) Task=`b4a61679-e63d-45bc-8d8c-53031fdbca31`::ref 0 aborting False >Thread-71007::DEBUG::2014-09-25 17:44:28,376::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71007::DEBUG::2014-09-25 17:44:28,376::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:28,390::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:28,391::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71008::DEBUG::2014-09-25 17:44:28,391::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71008::DEBUG::2014-09-25 17:44:28,393::task::595::Storage.TaskManager.Task::(_updateState) Task=`ffbaf455-27ad-4b5f-86cf-d2b166f08500`::moving from state init -> state preparing >Thread-71008::INFO::2014-09-25 17:44:28,393::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71008::DEBUG::2014-09-25 17:44:28,394::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e8b54f08-ff6e-4e09-8e12-ac224379ce9a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71008::DEBUG::2014-09-25 17:44:28,394::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71008::DEBUG::2014-09-25 17:44:28,395::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71008::DEBUG::2014-09-25 17:44:28,395::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e8b54f08-ff6e-4e09-8e12-ac224379ce9a`::Granted request >Thread-71008::DEBUG::2014-09-25 17:44:28,396::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ffbaf455-27ad-4b5f-86cf-d2b166f08500`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71008::DEBUG::2014-09-25 17:44:28,396::task::993::Storage.TaskManager.Task::(_decref) Task=`ffbaf455-27ad-4b5f-86cf-d2b166f08500`::ref 1 aborting False >Thread-71008::INFO::2014-09-25 17:44:28,400::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71008::DEBUG::2014-09-25 17:44:28,400::task::1191::Storage.TaskManager.Task::(prepare) Task=`ffbaf455-27ad-4b5f-86cf-d2b166f08500`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71008::DEBUG::2014-09-25 17:44:28,401::task::595::Storage.TaskManager.Task::(_updateState) Task=`ffbaf455-27ad-4b5f-86cf-d2b166f08500`::moving from state preparing -> state finished >Thread-71008::DEBUG::2014-09-25 17:44:28,401::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71008::DEBUG::2014-09-25 17:44:28,401::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71008::DEBUG::2014-09-25 17:44:28,401::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71008::DEBUG::2014-09-25 17:44:28,402::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71008::DEBUG::2014-09-25 17:44:28,402::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71008::DEBUG::2014-09-25 17:44:28,402::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71008::DEBUG::2014-09-25 17:44:28,403::task::993::Storage.TaskManager.Task::(_decref) Task=`ffbaf455-27ad-4b5f-86cf-d2b166f08500`::ref 0 aborting False >Thread-71008::DEBUG::2014-09-25 17:44:28,403::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71008::DEBUG::2014-09-25 17:44:28,404::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:28,979::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:28,982::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71009::DEBUG::2014-09-25 17:44:28,984::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:44:29,110::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:29,110::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:29,110::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:29,110::__init__::232::IOProcess::(_processLogs) (109246) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:29,115::__init__::232::IOProcess::(_processLogs) (109246) Queuing response >Thread-79::DEBUG::2014-09-25 17:44:29,116::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:44:29,133::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00341983 s, 164 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:29,135::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:29,135::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:29,135::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:29,135::__init__::232::IOProcess::(_processLogs) (109247) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:29,138::__init__::232::IOProcess::(_processLogs) (109247) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:29,139::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:29,140::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:29,140::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:29,141::__init__::232::IOProcess::(_processLogs) (109248) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:29,141::__init__::232::IOProcess::(_processLogs) (109248) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:29,141::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:29,142::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:29,142::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:29,143::__init__::232::IOProcess::(_processLogs) (109249) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:29,143::__init__::232::IOProcess::(_processLogs) (109249) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:29,144::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:29,144::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:29,145::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:29,145::__init__::232::IOProcess::(_processLogs) (109250) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:29,146::__init__::232::IOProcess::(_processLogs) (109250) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:29,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:29,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192592 s, 53.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:44:31,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:31,230::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017718 s, 57.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:31,999::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:32,000::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71010::DEBUG::2014-09-25 17:44:32,002::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:33,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:33,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214438 s, 47.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:33,574::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:33,574::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:33,575::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:33,575::__init__::232::IOProcess::(_processLogs) (109251) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:33,581::__init__::232::IOProcess::(_processLogs) (109251) Queuing response >Thread-81::DEBUG::2014-09-25 17:44:33,582::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:44:33,603::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00714189 s, 51.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:33,604::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:33,604::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:33,605::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:33,605::__init__::232::IOProcess::(_processLogs) (109252) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:33,609::__init__::232::IOProcess::(_processLogs) (109252) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:35,017::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:35,018::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71011::DEBUG::2014-09-25 17:44:35,021::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:35,303::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:35,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134523 s, 76.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:35,949::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:35,949::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:35,950::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:35,950::__init__::232::IOProcess::(_processLogs) (109253) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:35,950::__init__::232::IOProcess::(_processLogs) (109253) Queuing response >Thread-80::DEBUG::2014-09-25 17:44:35,951::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:44:35,960::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000397748 s, 950 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:35,961::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:35,962::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:35,962::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:35,962::__init__::232::IOProcess::(_processLogs) (109254) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:35,963::__init__::232::IOProcess::(_processLogs) (109254) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:37,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:37,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0197584 s, 51.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:38,034::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:38,036::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71012::DEBUG::2014-09-25 17:44:38,038::task::595::Storage.TaskManager.Task::(_updateState) Task=`714a4e1a-5472-4a3e-8c7d-1add54344ab3`::moving from state init -> state preparing >Thread-71012::INFO::2014-09-25 17:44:38,039::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71012::INFO::2014-09-25 17:44:38,039::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00341983', 'lastCheck': '8.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000397748', 'lastCheck': '2.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00714189', 'lastCheck': '4.4', 'valid': True}} >Thread-71012::DEBUG::2014-09-25 17:44:38,039::task::1191::Storage.TaskManager.Task::(prepare) Task=`714a4e1a-5472-4a3e-8c7d-1add54344ab3`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00341983', 'lastCheck': '8.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000397748', 'lastCheck': '2.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00714189', 'lastCheck': '4.4', 'valid': True}} >Thread-71012::DEBUG::2014-09-25 17:44:38,040::task::595::Storage.TaskManager.Task::(_updateState) Task=`714a4e1a-5472-4a3e-8c7d-1add54344ab3`::moving from state preparing -> state finished >Thread-71012::DEBUG::2014-09-25 17:44:38,040::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71012::DEBUG::2014-09-25 17:44:38,040::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71012::DEBUG::2014-09-25 17:44:38,040::task::993::Storage.TaskManager.Task::(_decref) Task=`714a4e1a-5472-4a3e-8c7d-1add54344ab3`::ref 0 aborting False >Thread-71012::DEBUG::2014-09-25 17:44:38,043::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:38,063::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:38,064::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71013::DEBUG::2014-09-25 17:44:38,065::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:38,434::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:38,435::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71014::DEBUG::2014-09-25 17:44:38,436::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71014::DEBUG::2014-09-25 17:44:38,437::task::595::Storage.TaskManager.Task::(_updateState) Task=`fbba8b73-fe37-4b2f-a7e5-7acf61e18849`::moving from state init -> state preparing >Thread-71014::INFO::2014-09-25 17:44:38,438::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71014::INFO::2014-09-25 17:44:38,438::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71014::DEBUG::2014-09-25 17:44:38,439::task::1191::Storage.TaskManager.Task::(prepare) Task=`fbba8b73-fe37-4b2f-a7e5-7acf61e18849`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71014::DEBUG::2014-09-25 17:44:38,439::task::595::Storage.TaskManager.Task::(_updateState) Task=`fbba8b73-fe37-4b2f-a7e5-7acf61e18849`::moving from state preparing -> state finished >Thread-71014::DEBUG::2014-09-25 17:44:38,439::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71014::DEBUG::2014-09-25 17:44:38,439::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71014::DEBUG::2014-09-25 17:44:38,439::task::993::Storage.TaskManager.Task::(_decref) Task=`fbba8b73-fe37-4b2f-a7e5-7acf61e18849`::ref 0 aborting False >Thread-71014::DEBUG::2014-09-25 17:44:38,440::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71014::DEBUG::2014-09-25 17:44:38,440::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:38,453::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71015::DEBUG::2014-09-25 17:44:38,454::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:44:38,454::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71015::DEBUG::2014-09-25 17:44:38,456::task::595::Storage.TaskManager.Task::(_updateState) Task=`87a9f5d2-0497-4e73-ad55-7e27b6e44517`::moving from state init -> state preparing >Thread-71015::INFO::2014-09-25 17:44:38,457::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71015::DEBUG::2014-09-25 17:44:38,457::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c48c1d98-2c3a-4bd4-9a2f-cc08da92fc03`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71015::DEBUG::2014-09-25 17:44:38,458::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71015::DEBUG::2014-09-25 17:44:38,458::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71015::DEBUG::2014-09-25 17:44:38,458::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c48c1d98-2c3a-4bd4-9a2f-cc08da92fc03`::Granted request >Thread-71015::DEBUG::2014-09-25 17:44:38,459::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`87a9f5d2-0497-4e73-ad55-7e27b6e44517`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71015::DEBUG::2014-09-25 17:44:38,459::task::993::Storage.TaskManager.Task::(_decref) Task=`87a9f5d2-0497-4e73-ad55-7e27b6e44517`::ref 1 aborting False >Thread-71015::INFO::2014-09-25 17:44:38,463::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71015::DEBUG::2014-09-25 17:44:38,464::task::1191::Storage.TaskManager.Task::(prepare) Task=`87a9f5d2-0497-4e73-ad55-7e27b6e44517`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71015::DEBUG::2014-09-25 17:44:38,464::task::595::Storage.TaskManager.Task::(_updateState) Task=`87a9f5d2-0497-4e73-ad55-7e27b6e44517`::moving from state preparing -> state finished >Thread-71015::DEBUG::2014-09-25 17:44:38,464::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71015::DEBUG::2014-09-25 17:44:38,464::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71015::DEBUG::2014-09-25 17:44:38,465::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71015::DEBUG::2014-09-25 17:44:38,465::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71015::DEBUG::2014-09-25 17:44:38,465::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71015::DEBUG::2014-09-25 17:44:38,465::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71015::DEBUG::2014-09-25 17:44:38,466::task::993::Storage.TaskManager.Task::(_decref) Task=`87a9f5d2-0497-4e73-ad55-7e27b6e44517`::ref 0 aborting False >Thread-71015::DEBUG::2014-09-25 17:44:38,466::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71015::DEBUG::2014-09-25 17:44:38,467::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:44:39,146::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:39,146::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:39,147::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:39,147::__init__::232::IOProcess::(_processLogs) (109255) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:39,150::__init__::232::IOProcess::(_processLogs) (109255) Queuing response >Thread-79::DEBUG::2014-09-25 17:44:39,151::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:44:39,166::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00607201 s, 92.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:39,167::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:39,168::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:39,168::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:39,168::__init__::232::IOProcess::(_processLogs) (109256) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:39,170::__init__::232::IOProcess::(_processLogs) (109256) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:39,172::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:39,172::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:39,173::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:39,173::__init__::232::IOProcess::(_processLogs) (109257) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:39,173::__init__::232::IOProcess::(_processLogs) (109257) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:39,174::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:39,174::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:39,175::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:39,175::__init__::232::IOProcess::(_processLogs) (109258) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:39,176::__init__::232::IOProcess::(_processLogs) (109258) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:39,176::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:39,176::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:39,177::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:39,178::__init__::232::IOProcess::(_processLogs) (109259) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:39,178::__init__::232::IOProcess::(_processLogs) (109259) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:39,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:39,422::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160406 s, 63.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:41,124::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:41,126::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71016::DEBUG::2014-09-25 17:44:41,128::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:41,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:41,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137478 s, 74.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:44:43,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:43,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159785 s, 64.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:43,611::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:43,612::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:43,612::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:43,612::__init__::232::IOProcess::(_processLogs) (109260) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:43,617::__init__::232::IOProcess::(_processLogs) (109260) Queuing response >Thread-81::DEBUG::2014-09-25 17:44:43,618::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:44:43,633::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00504905 s, 73.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:43,634::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:43,635::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:43,635::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:43,635::__init__::232::IOProcess::(_processLogs) (109261) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:43,637::__init__::232::IOProcess::(_processLogs) (109261) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:44,144::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:44,145::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71017::DEBUG::2014-09-25 17:44:44,148::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:45,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:45,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016939 s, 60.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:45,965::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:45,965::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:45,966::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:45,966::__init__::232::IOProcess::(_processLogs) (109262) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:45,966::__init__::232::IOProcess::(_processLogs) (109262) Queuing response >Thread-80::DEBUG::2014-09-25 17:44:45,967::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:44:45,976::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000402547 s, 939 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:45,977::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:45,978::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:45,978::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:45,978::__init__::232::IOProcess::(_processLogs) (109263) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:45,979::__init__::232::IOProcess::(_processLogs) (109263) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:47,163::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:47,164::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71018::DEBUG::2014-09-25 17:44:47,165::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:47,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:47,632::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015002 s, 68.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:48,494::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71019::DEBUG::2014-09-25 17:44:48,495::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:44:48,495::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71019::DEBUG::2014-09-25 17:44:48,498::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c680adf-e6ef-43b1-907b-f0e01e08ef40`::moving from state init -> state preparing >Thread-71019::INFO::2014-09-25 17:44:48,498::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71019::INFO::2014-09-25 17:44:48,498::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71019::DEBUG::2014-09-25 17:44:48,499::task::1191::Storage.TaskManager.Task::(prepare) Task=`3c680adf-e6ef-43b1-907b-f0e01e08ef40`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71019::DEBUG::2014-09-25 17:44:48,499::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c680adf-e6ef-43b1-907b-f0e01e08ef40`::moving from state preparing -> state finished >Thread-71019::DEBUG::2014-09-25 17:44:48,499::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71019::DEBUG::2014-09-25 17:44:48,499::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71019::DEBUG::2014-09-25 17:44:48,500::task::993::Storage.TaskManager.Task::(_decref) Task=`3c680adf-e6ef-43b1-907b-f0e01e08ef40`::ref 0 aborting False >Thread-71019::DEBUG::2014-09-25 17:44:48,500::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71019::DEBUG::2014-09-25 17:44:48,500::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:48,514::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71020::DEBUG::2014-09-25 17:44:48,515::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:44:48,515::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71020::DEBUG::2014-09-25 17:44:48,517::task::595::Storage.TaskManager.Task::(_updateState) Task=`342056f0-e2af-4b80-a462-e8a073043749`::moving from state init -> state preparing >Thread-71020::INFO::2014-09-25 17:44:48,517::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71020::DEBUG::2014-09-25 17:44:48,518::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`bddf9049-8035-4c6e-8ec2-a32744affa4e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71020::DEBUG::2014-09-25 17:44:48,518::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71020::DEBUG::2014-09-25 17:44:48,519::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71020::DEBUG::2014-09-25 17:44:48,519::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`bddf9049-8035-4c6e-8ec2-a32744affa4e`::Granted request >Thread-71020::DEBUG::2014-09-25 17:44:48,520::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`342056f0-e2af-4b80-a462-e8a073043749`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71020::DEBUG::2014-09-25 17:44:48,520::task::993::Storage.TaskManager.Task::(_decref) Task=`342056f0-e2af-4b80-a462-e8a073043749`::ref 1 aborting False >Thread-71020::INFO::2014-09-25 17:44:48,524::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71020::DEBUG::2014-09-25 17:44:48,524::task::1191::Storage.TaskManager.Task::(prepare) Task=`342056f0-e2af-4b80-a462-e8a073043749`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71020::DEBUG::2014-09-25 17:44:48,525::task::595::Storage.TaskManager.Task::(_updateState) Task=`342056f0-e2af-4b80-a462-e8a073043749`::moving from state preparing -> state finished >Thread-71020::DEBUG::2014-09-25 17:44:48,525::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71020::DEBUG::2014-09-25 17:44:48,525::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71020::DEBUG::2014-09-25 17:44:48,525::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71020::DEBUG::2014-09-25 17:44:48,526::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71020::DEBUG::2014-09-25 17:44:48,526::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71020::DEBUG::2014-09-25 17:44:48,526::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71020::DEBUG::2014-09-25 17:44:48,526::task::993::Storage.TaskManager.Task::(_decref) Task=`342056f0-e2af-4b80-a462-e8a073043749`::ref 0 aborting False >Thread-71020::DEBUG::2014-09-25 17:44:48,527::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71020::DEBUG::2014-09-25 17:44:48,527::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:44:49,179::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:49,179::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:49,179::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:49,180::__init__::232::IOProcess::(_processLogs) (109264) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:49,180::__init__::232::IOProcess::(_processLogs) (109264) Queuing response >Thread-79::DEBUG::2014-09-25 17:44:49,181::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:44:49,189::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000461224 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:49,191::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:49,191::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:49,191::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:49,191::__init__::232::IOProcess::(_processLogs) (109265) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:49,195::__init__::232::IOProcess::(_processLogs) (109265) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:49,197::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:49,197::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:49,197::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:49,197::__init__::232::IOProcess::(_processLogs) (109266) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:49,198::__init__::232::IOProcess::(_processLogs) (109266) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:49,199::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:49,199::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:49,199::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:49,200::__init__::232::IOProcess::(_processLogs) (109267) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:49,204::__init__::232::IOProcess::(_processLogs) (109267) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:49,204::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:49,205::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:49,205::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:49,205::__init__::232::IOProcess::(_processLogs) (109268) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:49,206::__init__::232::IOProcess::(_processLogs) (109268) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:49,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:49,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159607 s, 64.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:50,182::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:50,183::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71021::DEBUG::2014-09-25 17:44:50,185::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:51,700::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:51,723::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136088 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:53,200::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:53,201::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71022::DEBUG::2014-09-25 17:44:53,204::task::595::Storage.TaskManager.Task::(_updateState) Task=`1025950b-27c8-4bc5-9770-f41b7cb8bae0`::moving from state init -> state preparing >Thread-71022::INFO::2014-09-25 17:44:53,204::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71022::INFO::2014-09-25 17:44:53,205::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000461224', 'lastCheck': '4.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000402547', 'lastCheck': '7.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00504905', 'lastCheck': '9.6', 'valid': True}} >Thread-71022::DEBUG::2014-09-25 17:44:53,205::task::1191::Storage.TaskManager.Task::(prepare) Task=`1025950b-27c8-4bc5-9770-f41b7cb8bae0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000461224', 'lastCheck': '4.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000402547', 'lastCheck': '7.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00504905', 'lastCheck': '9.6', 'valid': True}} >Thread-71022::DEBUG::2014-09-25 17:44:53,205::task::595::Storage.TaskManager.Task::(_updateState) Task=`1025950b-27c8-4bc5-9770-f41b7cb8bae0`::moving from state preparing -> state finished >Thread-71022::DEBUG::2014-09-25 17:44:53,205::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71022::DEBUG::2014-09-25 17:44:53,205::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71022::DEBUG::2014-09-25 17:44:53,206::task::993::Storage.TaskManager.Task::(_decref) Task=`1025950b-27c8-4bc5-9770-f41b7cb8bae0`::ref 0 aborting False >Thread-71022::DEBUG::2014-09-25 17:44:53,208::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:53,228::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:53,230::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71023::DEBUG::2014-09-25 17:44:53,232::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:44:53,638::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:53,639::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:53,639::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:53,639::__init__::232::IOProcess::(_processLogs) (109269) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:53,642::__init__::232::IOProcess::(_processLogs) (109269) Queuing response >Thread-81::DEBUG::2014-09-25 17:44:53,643::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:44:53,657::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00392863 s, 94.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:53,658::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:53,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:53,658::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:53,659::__init__::232::IOProcess::(_processLogs) (109270) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:53,661::__init__::232::IOProcess::(_processLogs) (109270) Queuing response >Dummy-148::DEBUG::2014-09-25 17:44:53,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:53,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0196698 s, 52.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:44:55,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:55,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207595 s, 49.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:55,981::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:55,982::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:55,982::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:55,982::__init__::232::IOProcess::(_processLogs) (109271) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:55,983::__init__::232::IOProcess::(_processLogs) (109271) Queuing response >Thread-80::DEBUG::2014-09-25 17:44:55,984::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:44:55,992::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000339078 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:55,993::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:55,994::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:55,994::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:55,996::__init__::232::IOProcess::(_processLogs) (109272) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:55,996::__init__::232::IOProcess::(_processLogs) (109272) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:56,288::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:56,289::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71024::DEBUG::2014-09-25 17:44:56,291::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:57,848::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:57,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152673 s, 67.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:58,557::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:58,558::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71025::DEBUG::2014-09-25 17:44:58,558::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71025::DEBUG::2014-09-25 17:44:58,560::task::595::Storage.TaskManager.Task::(_updateState) Task=`57927c04-b0dc-455f-98a3-11f0beaf6bb7`::moving from state init -> state preparing >Thread-71025::INFO::2014-09-25 17:44:58,561::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71025::INFO::2014-09-25 17:44:58,561::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71025::DEBUG::2014-09-25 17:44:58,561::task::1191::Storage.TaskManager.Task::(prepare) Task=`57927c04-b0dc-455f-98a3-11f0beaf6bb7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71025::DEBUG::2014-09-25 17:44:58,562::task::595::Storage.TaskManager.Task::(_updateState) Task=`57927c04-b0dc-455f-98a3-11f0beaf6bb7`::moving from state preparing -> state finished >Thread-71025::DEBUG::2014-09-25 17:44:58,562::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71025::DEBUG::2014-09-25 17:44:58,562::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71025::DEBUG::2014-09-25 17:44:58,562::task::993::Storage.TaskManager.Task::(_decref) Task=`57927c04-b0dc-455f-98a3-11f0beaf6bb7`::ref 0 aborting False >Thread-71025::DEBUG::2014-09-25 17:44:58,563::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71025::DEBUG::2014-09-25 17:44:58,563::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:58,577::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71026::DEBUG::2014-09-25 17:44:58,578::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:44:58,578::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71026::DEBUG::2014-09-25 17:44:58,580::task::595::Storage.TaskManager.Task::(_updateState) Task=`f7e187fd-cd6b-4b0a-873f-5578230a0a60`::moving from state init -> state preparing >Thread-71026::INFO::2014-09-25 17:44:58,581::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71026::DEBUG::2014-09-25 17:44:58,581::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0a65ceb4-9769-430c-8952-ca389f2b7aea`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71026::DEBUG::2014-09-25 17:44:58,582::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71026::DEBUG::2014-09-25 17:44:58,582::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71026::DEBUG::2014-09-25 17:44:58,582::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`0a65ceb4-9769-430c-8952-ca389f2b7aea`::Granted request >Thread-71026::DEBUG::2014-09-25 17:44:58,584::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`f7e187fd-cd6b-4b0a-873f-5578230a0a60`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71026::DEBUG::2014-09-25 17:44:58,584::task::993::Storage.TaskManager.Task::(_decref) Task=`f7e187fd-cd6b-4b0a-873f-5578230a0a60`::ref 1 aborting False >Thread-71026::INFO::2014-09-25 17:44:58,588::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71026::DEBUG::2014-09-25 17:44:58,589::task::1191::Storage.TaskManager.Task::(prepare) Task=`f7e187fd-cd6b-4b0a-873f-5578230a0a60`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71026::DEBUG::2014-09-25 17:44:58,589::task::595::Storage.TaskManager.Task::(_updateState) Task=`f7e187fd-cd6b-4b0a-873f-5578230a0a60`::moving from state preparing -> state finished >Thread-71026::DEBUG::2014-09-25 17:44:58,589::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71026::DEBUG::2014-09-25 17:44:58,589::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71026::DEBUG::2014-09-25 17:44:58,590::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71026::DEBUG::2014-09-25 17:44:58,590::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71026::DEBUG::2014-09-25 17:44:58,590::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71026::DEBUG::2014-09-25 17:44:58,591::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71026::DEBUG::2014-09-25 17:44:58,591::task::993::Storage.TaskManager.Task::(_decref) Task=`f7e187fd-cd6b-4b0a-873f-5578230a0a60`::ref 0 aborting False >Thread-71026::DEBUG::2014-09-25 17:44:58,591::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71026::DEBUG::2014-09-25 17:44:58,592::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:44:59,208::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:59,209::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:59,209::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:59,209::__init__::232::IOProcess::(_processLogs) (109273) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:59,212::__init__::232::IOProcess::(_processLogs) (109273) Queuing response >Thread-79::DEBUG::2014-09-25 17:44:59,213::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:44:59,237::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00704091 s, 79.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:44:59,238::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:59,239::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:59,239::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:59,239::__init__::232::IOProcess::(_processLogs) (109274) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:44:59,245::__init__::232::IOProcess::(_processLogs) (109274) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:59,246::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:59,247::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:59,247::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:59,248::__init__::232::IOProcess::(_processLogs) (109275) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:59,248::__init__::232::IOProcess::(_processLogs) (109275) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:59,249::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:59,249::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:59,249::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:59,250::__init__::232::IOProcess::(_processLogs) (109276) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:59,250::__init__::232::IOProcess::(_processLogs) (109276) Queuing response >Thread-29::DEBUG::2014-09-25 17:44:59,251::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:44:59,251::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:44:59,252::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:44:59,253::__init__::232::IOProcess::(_processLogs) (109277) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:44:59,253::__init__::232::IOProcess::(_processLogs) (109277) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:44:59,307::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:44:59,308::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71027::DEBUG::2014-09-25 17:44:59,310::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:44:59,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:44:59,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157914 s, 64.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:45:01,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:01,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021619 s, 47.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:02,325::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:02,327::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71028::DEBUG::2014-09-25 17:45:02,329::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:45:03,663::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:03,664::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:03,664::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:03,664::__init__::232::IOProcess::(_processLogs) (109278) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:03,671::__init__::232::IOProcess::(_processLogs) (109278) Queuing response >Thread-81::DEBUG::2014-09-25 17:45:03,672::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:45:03,690::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00314377 s, 118 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:03,691::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:03,692::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:03,692::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:03,692::__init__::232::IOProcess::(_processLogs) (109279) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:03,696::__init__::232::IOProcess::(_processLogs) (109279) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:03,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:04,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177374 s, 57.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:05,346::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:05,347::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71029::DEBUG::2014-09-25 17:45:05,350::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:45:05,996::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:05,997::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:05,997::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:05,997::__init__::232::IOProcess::(_processLogs) (109280) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:05,998::__init__::232::IOProcess::(_processLogs) (109280) Queuing response >Thread-80::DEBUG::2014-09-25 17:45:05,999::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:45:06,008::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000550795 s, 686 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:06,009::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:06,010::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:06,010::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:06,010::__init__::232::IOProcess::(_processLogs) (109281) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:06,011::__init__::232::IOProcess::(_processLogs) (109281) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:06,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:06,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0217609 s, 47.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:45:08,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:08,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0255675 s, 40.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:08,366::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:08,368::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71030::DEBUG::2014-09-25 17:45:08,370::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d1da0f4-4288-4217-9d04-effc40abe7b3`::moving from state init -> state preparing >Thread-71030::INFO::2014-09-25 17:45:08,371::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71030::INFO::2014-09-25 17:45:08,371::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00704091', 'lastCheck': '9.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000550795', 'lastCheck': '2.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00314377', 'lastCheck': '4.7', 'valid': True}} >Thread-71030::DEBUG::2014-09-25 17:45:08,371::task::1191::Storage.TaskManager.Task::(prepare) Task=`2d1da0f4-4288-4217-9d04-effc40abe7b3`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00704091', 'lastCheck': '9.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000550795', 'lastCheck': '2.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00314377', 'lastCheck': '4.7', 'valid': True}} >Thread-71030::DEBUG::2014-09-25 17:45:08,372::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d1da0f4-4288-4217-9d04-effc40abe7b3`::moving from state preparing -> state finished >Thread-71030::DEBUG::2014-09-25 17:45:08,372::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71030::DEBUG::2014-09-25 17:45:08,372::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71030::DEBUG::2014-09-25 17:45:08,372::task::993::Storage.TaskManager.Task::(_decref) Task=`2d1da0f4-4288-4217-9d04-effc40abe7b3`::ref 0 aborting False >Thread-71030::DEBUG::2014-09-25 17:45:08,375::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:08,395::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:08,396::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71031::DEBUG::2014-09-25 17:45:08,398::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:08,621::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:08,622::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71032::DEBUG::2014-09-25 17:45:08,622::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71032::DEBUG::2014-09-25 17:45:08,624::task::595::Storage.TaskManager.Task::(_updateState) Task=`cb9c125d-4dc0-4ce6-8f6e-16b6b624f7eb`::moving from state init -> state preparing >Thread-71032::INFO::2014-09-25 17:45:08,624::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71032::INFO::2014-09-25 17:45:08,625::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71032::DEBUG::2014-09-25 17:45:08,625::task::1191::Storage.TaskManager.Task::(prepare) Task=`cb9c125d-4dc0-4ce6-8f6e-16b6b624f7eb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71032::DEBUG::2014-09-25 17:45:08,625::task::595::Storage.TaskManager.Task::(_updateState) Task=`cb9c125d-4dc0-4ce6-8f6e-16b6b624f7eb`::moving from state preparing -> state finished >Thread-71032::DEBUG::2014-09-25 17:45:08,626::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71032::DEBUG::2014-09-25 17:45:08,626::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71032::DEBUG::2014-09-25 17:45:08,626::task::993::Storage.TaskManager.Task::(_decref) Task=`cb9c125d-4dc0-4ce6-8f6e-16b6b624f7eb`::ref 0 aborting False >Thread-71032::DEBUG::2014-09-25 17:45:08,626::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71032::DEBUG::2014-09-25 17:45:08,627::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:08,639::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:08,640::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71033::DEBUG::2014-09-25 17:45:08,640::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71033::DEBUG::2014-09-25 17:45:08,642::task::595::Storage.TaskManager.Task::(_updateState) Task=`526c63e6-f11d-4e0a-b8f4-93acc70de062`::moving from state init -> state preparing >Thread-71033::INFO::2014-09-25 17:45:08,642::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71033::DEBUG::2014-09-25 17:45:08,643::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1bdd28f4-c3d8-4036-8c41-dd7a90e28696`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71033::DEBUG::2014-09-25 17:45:08,643::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71033::DEBUG::2014-09-25 17:45:08,644::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71033::DEBUG::2014-09-25 17:45:08,644::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1bdd28f4-c3d8-4036-8c41-dd7a90e28696`::Granted request >Thread-71033::DEBUG::2014-09-25 17:45:08,645::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`526c63e6-f11d-4e0a-b8f4-93acc70de062`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71033::DEBUG::2014-09-25 17:45:08,645::task::993::Storage.TaskManager.Task::(_decref) Task=`526c63e6-f11d-4e0a-b8f4-93acc70de062`::ref 1 aborting False >Thread-71033::INFO::2014-09-25 17:45:08,649::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71033::DEBUG::2014-09-25 17:45:08,649::task::1191::Storage.TaskManager.Task::(prepare) Task=`526c63e6-f11d-4e0a-b8f4-93acc70de062`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71033::DEBUG::2014-09-25 17:45:08,650::task::595::Storage.TaskManager.Task::(_updateState) Task=`526c63e6-f11d-4e0a-b8f4-93acc70de062`::moving from state preparing -> state finished >Thread-71033::DEBUG::2014-09-25 17:45:08,650::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71033::DEBUG::2014-09-25 17:45:08,650::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71033::DEBUG::2014-09-25 17:45:08,651::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71033::DEBUG::2014-09-25 17:45:08,651::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71033::DEBUG::2014-09-25 17:45:08,651::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71033::DEBUG::2014-09-25 17:45:08,651::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71033::DEBUG::2014-09-25 17:45:08,652::task::993::Storage.TaskManager.Task::(_decref) Task=`526c63e6-f11d-4e0a-b8f4-93acc70de062`::ref 0 aborting False >Thread-71033::DEBUG::2014-09-25 17:45:08,652::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71033::DEBUG::2014-09-25 17:45:08,653::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:45:09,254::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:09,254::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:09,254::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:09,254::__init__::232::IOProcess::(_processLogs) (109282) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:09,255::__init__::232::IOProcess::(_processLogs) (109282) Queuing response >Thread-79::DEBUG::2014-09-25 17:45:09,256::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:45:09,265::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000400682 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:09,267::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:09,267::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:09,268::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:09,269::__init__::232::IOProcess::(_processLogs) (109283) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:09,269::__init__::232::IOProcess::(_processLogs) (109283) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:09,269::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:09,270::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:09,270::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:09,270::__init__::232::IOProcess::(_processLogs) (109284) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:09,270::__init__::232::IOProcess::(_processLogs) (109284) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:09,271::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:09,272::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:09,272::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:09,272::__init__::232::IOProcess::(_processLogs) (109285) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:09,272::__init__::232::IOProcess::(_processLogs) (109285) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:09,273::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:09,273::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:09,274::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:09,274::__init__::232::IOProcess::(_processLogs) (109286) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:09,274::__init__::232::IOProcess::(_processLogs) (109286) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:10,186::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:10,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0566038 s, 18.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:11,459::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:11,460::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71034::DEBUG::2014-09-25 17:45:11,463::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:12,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:12,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142743 s, 71.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:13,698::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:13,699::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:13,699::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:13,699::__init__::232::IOProcess::(_processLogs) (109287) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:13,707::__init__::232::IOProcess::(_processLogs) (109287) Queuing response >Thread-81::DEBUG::2014-09-25 17:45:13,709::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:45:13,727::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00138814 s, 267 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:13,728::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:13,728::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:13,729::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:13,730::__init__::232::IOProcess::(_processLogs) (109288) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:13,731::__init__::232::IOProcess::(_processLogs) (109288) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:14,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:14,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159076 s, 64.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:14,478::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:14,479::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71035::DEBUG::2014-09-25 17:45:14,482::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:45:16,013::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:16,013::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:16,013::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:16,014::__init__::232::IOProcess::(_processLogs) (109289) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:16,014::__init__::232::IOProcess::(_processLogs) (109289) Queuing response >Thread-80::DEBUG::2014-09-25 17:45:16,016::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:45:16,026::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000435231 s, 869 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:16,029::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:16,030::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:16,031::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:16,032::__init__::232::IOProcess::(_processLogs) (109290) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:16,034::__init__::232::IOProcess::(_processLogs) (109290) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:16,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:16,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0348858 s, 29.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:17,497::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:17,499::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71036::DEBUG::2014-09-25 17:45:17,501::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:18,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:18,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.022121 s, 46.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:18,681::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71037::DEBUG::2014-09-25 17:45:18,683::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:45:18,683::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71037::DEBUG::2014-09-25 17:45:18,686::task::595::Storage.TaskManager.Task::(_updateState) Task=`bfed4ee4-d3f3-4716-8919-511a20d5bba2`::moving from state init -> state preparing >Thread-71037::INFO::2014-09-25 17:45:18,686::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71037::INFO::2014-09-25 17:45:18,686::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71037::DEBUG::2014-09-25 17:45:18,687::task::1191::Storage.TaskManager.Task::(prepare) Task=`bfed4ee4-d3f3-4716-8919-511a20d5bba2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71037::DEBUG::2014-09-25 17:45:18,687::task::595::Storage.TaskManager.Task::(_updateState) Task=`bfed4ee4-d3f3-4716-8919-511a20d5bba2`::moving from state preparing -> state finished >Thread-71037::DEBUG::2014-09-25 17:45:18,687::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71037::DEBUG::2014-09-25 17:45:18,687::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71037::DEBUG::2014-09-25 17:45:18,688::task::993::Storage.TaskManager.Task::(_decref) Task=`bfed4ee4-d3f3-4716-8919-511a20d5bba2`::ref 0 aborting False >Thread-71037::DEBUG::2014-09-25 17:45:18,688::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71037::DEBUG::2014-09-25 17:45:18,688::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:18,701::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71038::DEBUG::2014-09-25 17:45:18,703::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:45:18,703::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71038::DEBUG::2014-09-25 17:45:18,705::task::595::Storage.TaskManager.Task::(_updateState) Task=`0df5d19d-aeef-466d-a18e-35b83da2de25`::moving from state init -> state preparing >Thread-71038::INFO::2014-09-25 17:45:18,705::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71038::DEBUG::2014-09-25 17:45:18,706::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ee81edfd-7460-4892-8a8a-0820bd7c9637`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71038::DEBUG::2014-09-25 17:45:18,706::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71038::DEBUG::2014-09-25 17:45:18,706::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71038::DEBUG::2014-09-25 17:45:18,707::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ee81edfd-7460-4892-8a8a-0820bd7c9637`::Granted request >Thread-71038::DEBUG::2014-09-25 17:45:18,707::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0df5d19d-aeef-466d-a18e-35b83da2de25`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71038::DEBUG::2014-09-25 17:45:18,707::task::993::Storage.TaskManager.Task::(_decref) Task=`0df5d19d-aeef-466d-a18e-35b83da2de25`::ref 1 aborting False >Thread-71038::INFO::2014-09-25 17:45:18,712::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71038::DEBUG::2014-09-25 17:45:18,712::task::1191::Storage.TaskManager.Task::(prepare) Task=`0df5d19d-aeef-466d-a18e-35b83da2de25`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71038::DEBUG::2014-09-25 17:45:18,712::task::595::Storage.TaskManager.Task::(_updateState) Task=`0df5d19d-aeef-466d-a18e-35b83da2de25`::moving from state preparing -> state finished >Thread-71038::DEBUG::2014-09-25 17:45:18,713::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71038::DEBUG::2014-09-25 17:45:18,713::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71038::DEBUG::2014-09-25 17:45:18,713::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71038::DEBUG::2014-09-25 17:45:18,713::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71038::DEBUG::2014-09-25 17:45:18,714::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71038::DEBUG::2014-09-25 17:45:18,714::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71038::DEBUG::2014-09-25 17:45:18,714::task::993::Storage.TaskManager.Task::(_decref) Task=`0df5d19d-aeef-466d-a18e-35b83da2de25`::ref 0 aborting False >Thread-71038::DEBUG::2014-09-25 17:45:18,715::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71038::DEBUG::2014-09-25 17:45:18,715::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:45:19,276::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:19,277::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:19,277::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:19,277::__init__::232::IOProcess::(_processLogs) (109291) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:19,278::__init__::232::IOProcess::(_processLogs) (109291) Queuing response >Thread-79::DEBUG::2014-09-25 17:45:19,279::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:45:19,288::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000459863 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:19,289::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:19,290::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:19,290::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:19,291::__init__::232::IOProcess::(_processLogs) (109292) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:19,292::__init__::232::IOProcess::(_processLogs) (109292) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:19,294::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:19,294::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:19,294::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:19,294::__init__::232::IOProcess::(_processLogs) (109293) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:19,295::__init__::232::IOProcess::(_processLogs) (109293) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:19,296::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:19,296::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:19,296::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:19,297::__init__::232::IOProcess::(_processLogs) (109294) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:19,297::__init__::232::IOProcess::(_processLogs) (109294) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:19,298::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:19,298::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:19,298::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:19,299::__init__::232::IOProcess::(_processLogs) (109295) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:19,299::__init__::232::IOProcess::(_processLogs) (109295) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:20,516::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:20,518::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71039::DEBUG::2014-09-25 17:45:20,520::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:20,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:20,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184753 s, 55.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:45:22,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:22,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159343 s, 64.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:23,534::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:23,536::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71040::DEBUG::2014-09-25 17:45:23,538::task::595::Storage.TaskManager.Task::(_updateState) Task=`94010153-ac54-4f63-bf6c-2a7edfde4abb`::moving from state init -> state preparing >Thread-71040::INFO::2014-09-25 17:45:23,539::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71040::INFO::2014-09-25 17:45:23,539::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000459863', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000435231', 'lastCheck': '7.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00138814', 'lastCheck': '9.8', 'valid': True}} >Thread-71040::DEBUG::2014-09-25 17:45:23,539::task::1191::Storage.TaskManager.Task::(prepare) Task=`94010153-ac54-4f63-bf6c-2a7edfde4abb`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000459863', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000435231', 'lastCheck': '7.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00138814', 'lastCheck': '9.8', 'valid': True}} >Thread-71040::DEBUG::2014-09-25 17:45:23,540::task::595::Storage.TaskManager.Task::(_updateState) Task=`94010153-ac54-4f63-bf6c-2a7edfde4abb`::moving from state preparing -> state finished >Thread-71040::DEBUG::2014-09-25 17:45:23,540::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71040::DEBUG::2014-09-25 17:45:23,540::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71040::DEBUG::2014-09-25 17:45:23,540::task::993::Storage.TaskManager.Task::(_decref) Task=`94010153-ac54-4f63-bf6c-2a7edfde4abb`::ref 0 aborting False >Thread-71040::DEBUG::2014-09-25 17:45:23,543::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:23,562::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:23,563::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71041::DEBUG::2014-09-25 17:45:23,565::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:45:23,731::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:23,732::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:23,732::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:23,732::__init__::232::IOProcess::(_processLogs) (109296) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:23,735::__init__::232::IOProcess::(_processLogs) (109296) Queuing response >Thread-81::DEBUG::2014-09-25 17:45:23,736::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:45:23,760::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00543412 s, 68.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:23,761::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:23,762::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:23,762::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:23,762::__init__::232::IOProcess::(_processLogs) (109297) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:23,764::__init__::232::IOProcess::(_processLogs) (109297) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:24,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:24,777::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133672 s, 76.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:26,037::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:26,038::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:26,038::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:26,038::__init__::232::IOProcess::(_processLogs) (109298) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:26,039::__init__::232::IOProcess::(_processLogs) (109298) Queuing response >Thread-80::DEBUG::2014-09-25 17:45:26,040::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:45:26,049::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000436049 s, 867 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:26,050::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:26,050::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:26,051::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:26,051::__init__::232::IOProcess::(_processLogs) (109299) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:26,051::__init__::232::IOProcess::(_processLogs) (109299) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:26,623::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:26,624::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71042::DEBUG::2014-09-25 17:45:26,626::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:26,797::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:26,826::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162018 s, 63.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:28,742::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71043::DEBUG::2014-09-25 17:45:28,743::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:45:28,743::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71043::DEBUG::2014-09-25 17:45:28,746::task::595::Storage.TaskManager.Task::(_updateState) Task=`3305e9ad-f27e-4be1-b200-88c9f37c230b`::moving from state init -> state preparing >Thread-71043::INFO::2014-09-25 17:45:28,746::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71043::INFO::2014-09-25 17:45:28,746::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71043::DEBUG::2014-09-25 17:45:28,747::task::1191::Storage.TaskManager.Task::(prepare) Task=`3305e9ad-f27e-4be1-b200-88c9f37c230b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71043::DEBUG::2014-09-25 17:45:28,747::task::595::Storage.TaskManager.Task::(_updateState) Task=`3305e9ad-f27e-4be1-b200-88c9f37c230b`::moving from state preparing -> state finished >Thread-71043::DEBUG::2014-09-25 17:45:28,747::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71043::DEBUG::2014-09-25 17:45:28,747::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71043::DEBUG::2014-09-25 17:45:28,747::task::993::Storage.TaskManager.Task::(_decref) Task=`3305e9ad-f27e-4be1-b200-88c9f37c230b`::ref 0 aborting False >Thread-71043::DEBUG::2014-09-25 17:45:28,748::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71043::DEBUG::2014-09-25 17:45:28,748::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:28,761::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:28,762::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71044::DEBUG::2014-09-25 17:45:28,762::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71044::DEBUG::2014-09-25 17:45:28,764::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e1589c3-449f-42f5-a94b-0a074a116bbb`::moving from state init -> state preparing >Thread-71044::INFO::2014-09-25 17:45:28,764::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71044::DEBUG::2014-09-25 17:45:28,765::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2e73c071-899b-4cd9-8e05-fdef6a5ef416`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71044::DEBUG::2014-09-25 17:45:28,765::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71044::DEBUG::2014-09-25 17:45:28,766::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71044::DEBUG::2014-09-25 17:45:28,766::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2e73c071-899b-4cd9-8e05-fdef6a5ef416`::Granted request >Thread-71044::DEBUG::2014-09-25 17:45:28,766::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3e1589c3-449f-42f5-a94b-0a074a116bbb`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71044::DEBUG::2014-09-25 17:45:28,767::task::993::Storage.TaskManager.Task::(_decref) Task=`3e1589c3-449f-42f5-a94b-0a074a116bbb`::ref 1 aborting False >Thread-71044::INFO::2014-09-25 17:45:28,771::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71044::DEBUG::2014-09-25 17:45:28,771::task::1191::Storage.TaskManager.Task::(prepare) Task=`3e1589c3-449f-42f5-a94b-0a074a116bbb`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71044::DEBUG::2014-09-25 17:45:28,771::task::595::Storage.TaskManager.Task::(_updateState) Task=`3e1589c3-449f-42f5-a94b-0a074a116bbb`::moving from state preparing -> state finished >Thread-71044::DEBUG::2014-09-25 17:45:28,772::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71044::DEBUG::2014-09-25 17:45:28,772::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71044::DEBUG::2014-09-25 17:45:28,772::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71044::DEBUG::2014-09-25 17:45:28,773::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71044::DEBUG::2014-09-25 17:45:28,773::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71044::DEBUG::2014-09-25 17:45:28,773::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71044::DEBUG::2014-09-25 17:45:28,773::task::993::Storage.TaskManager.Task::(_decref) Task=`3e1589c3-449f-42f5-a94b-0a074a116bbb`::ref 0 aborting False >Thread-71044::DEBUG::2014-09-25 17:45:28,774::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71044::DEBUG::2014-09-25 17:45:28,774::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:28,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:45:29,301::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:29,302::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:29,302::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:29,302::__init__::232::IOProcess::(_processLogs) (109300) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:29,366::__init__::232::IOProcess::(_processLogs) (109300) Queuing response >Thread-79::DEBUG::2014-09-25 17:45:29,369::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:29,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0815576 s, 12.6 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 17:45:29,394::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00404195 s, 139 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:29,396::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:29,397::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:29,398::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:29,399::__init__::232::IOProcess::(_processLogs) (109301) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:29,403::__init__::232::IOProcess::(_processLogs) (109301) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:29,406::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:29,407::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:29,408::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:29,409::__init__::232::IOProcess::(_processLogs) (109302) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:29,410::__init__::232::IOProcess::(_processLogs) (109302) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:29,412::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:29,413::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:29,414::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:29,415::__init__::232::IOProcess::(_processLogs) (109303) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:29,416::__init__::232::IOProcess::(_processLogs) (109303) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:29,418::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:29,419::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:29,420::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:29,420::__init__::232::IOProcess::(_processLogs) (109304) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:29,422::__init__::232::IOProcess::(_processLogs) (109304) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:29,641::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:29,643::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71045::DEBUG::2014-09-25 17:45:29,645::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:31,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:31,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215281 s, 47.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:32,660::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:32,662::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71046::DEBUG::2014-09-25 17:45:32,664::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:33,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:33,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169345 s, 60.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:33,766::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:33,766::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:33,767::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:33,767::__init__::232::IOProcess::(_processLogs) (109305) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:33,772::__init__::232::IOProcess::(_processLogs) (109305) Queuing response >Thread-81::DEBUG::2014-09-25 17:45:33,773::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:45:33,785::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00185288 s, 200 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:33,786::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:33,786::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:33,787::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:33,787::__init__::232::IOProcess::(_processLogs) (109306) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:33,789::__init__::232::IOProcess::(_processLogs) (109306) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:35,533::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:35,565::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0198528 s, 51.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:35,680::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:35,681::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71047::DEBUG::2014-09-25 17:45:35,683::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:45:36,054::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:36,054::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:36,054::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:36,055::__init__::232::IOProcess::(_processLogs) (109307) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:36,055::__init__::232::IOProcess::(_processLogs) (109307) Queuing response >Thread-80::DEBUG::2014-09-25 17:45:36,056::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:45:36,065::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000514921 s, 734 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:36,066::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:36,066::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:36,067::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:36,067::__init__::232::IOProcess::(_processLogs) (109308) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:36,067::__init__::232::IOProcess::(_processLogs) (109308) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:37,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:37,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132092 s, 77.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:38,697::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:38,699::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71048::DEBUG::2014-09-25 17:45:38,701::task::595::Storage.TaskManager.Task::(_updateState) Task=`3937f920-d479-4007-9526-de1bf373318b`::moving from state init -> state preparing >Thread-71048::INFO::2014-09-25 17:45:38,702::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71048::INFO::2014-09-25 17:45:38,702::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00404195', 'lastCheck': '9.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000514921', 'lastCheck': '2.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00185288', 'lastCheck': '4.9', 'valid': True}} >Thread-71048::DEBUG::2014-09-25 17:45:38,703::task::1191::Storage.TaskManager.Task::(prepare) Task=`3937f920-d479-4007-9526-de1bf373318b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00404195', 'lastCheck': '9.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000514921', 'lastCheck': '2.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00185288', 'lastCheck': '4.9', 'valid': True}} >Thread-71048::DEBUG::2014-09-25 17:45:38,703::task::595::Storage.TaskManager.Task::(_updateState) Task=`3937f920-d479-4007-9526-de1bf373318b`::moving from state preparing -> state finished >Thread-71048::DEBUG::2014-09-25 17:45:38,703::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71048::DEBUG::2014-09-25 17:45:38,703::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71048::DEBUG::2014-09-25 17:45:38,703::task::993::Storage.TaskManager.Task::(_decref) Task=`3937f920-d479-4007-9526-de1bf373318b`::ref 0 aborting False >Thread-71048::DEBUG::2014-09-25 17:45:38,706::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:38,729::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:38,730::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71049::DEBUG::2014-09-25 17:45:38,732::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:38,802::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:38,803::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71050::DEBUG::2014-09-25 17:45:38,803::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71050::DEBUG::2014-09-25 17:45:38,805::task::595::Storage.TaskManager.Task::(_updateState) Task=`6e28f29d-42a6-4a06-9477-82b6552f2e70`::moving from state init -> state preparing >Thread-71050::INFO::2014-09-25 17:45:38,805::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71050::INFO::2014-09-25 17:45:38,806::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71050::DEBUG::2014-09-25 17:45:38,806::task::1191::Storage.TaskManager.Task::(prepare) Task=`6e28f29d-42a6-4a06-9477-82b6552f2e70`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71050::DEBUG::2014-09-25 17:45:38,806::task::595::Storage.TaskManager.Task::(_updateState) Task=`6e28f29d-42a6-4a06-9477-82b6552f2e70`::moving from state preparing -> state finished >Thread-71050::DEBUG::2014-09-25 17:45:38,806::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71050::DEBUG::2014-09-25 17:45:38,807::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71050::DEBUG::2014-09-25 17:45:38,807::task::993::Storage.TaskManager.Task::(_decref) Task=`6e28f29d-42a6-4a06-9477-82b6552f2e70`::ref 0 aborting False >Thread-71050::DEBUG::2014-09-25 17:45:38,807::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71050::DEBUG::2014-09-25 17:45:38,808::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:38,821::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:38,822::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71051::DEBUG::2014-09-25 17:45:38,823::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71051::DEBUG::2014-09-25 17:45:38,824::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ef1e51f-51fb-4589-866c-c2c45609e9bd`::moving from state init -> state preparing >Thread-71051::INFO::2014-09-25 17:45:38,824::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71051::DEBUG::2014-09-25 17:45:38,825::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`274f8d83-bbba-475b-b814-3d12bca10fb2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71051::DEBUG::2014-09-25 17:45:38,825::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71051::DEBUG::2014-09-25 17:45:38,826::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71051::DEBUG::2014-09-25 17:45:38,826::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`274f8d83-bbba-475b-b814-3d12bca10fb2`::Granted request >Thread-71051::DEBUG::2014-09-25 17:45:38,828::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0ef1e51f-51fb-4589-866c-c2c45609e9bd`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71051::DEBUG::2014-09-25 17:45:38,828::task::993::Storage.TaskManager.Task::(_decref) Task=`0ef1e51f-51fb-4589-866c-c2c45609e9bd`::ref 1 aborting False >Thread-71051::INFO::2014-09-25 17:45:38,832::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71051::DEBUG::2014-09-25 17:45:38,832::task::1191::Storage.TaskManager.Task::(prepare) Task=`0ef1e51f-51fb-4589-866c-c2c45609e9bd`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71051::DEBUG::2014-09-25 17:45:38,833::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ef1e51f-51fb-4589-866c-c2c45609e9bd`::moving from state preparing -> state finished >Thread-71051::DEBUG::2014-09-25 17:45:38,833::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71051::DEBUG::2014-09-25 17:45:38,833::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71051::DEBUG::2014-09-25 17:45:38,833::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71051::DEBUG::2014-09-25 17:45:38,834::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71051::DEBUG::2014-09-25 17:45:38,834::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71051::DEBUG::2014-09-25 17:45:38,834::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71051::DEBUG::2014-09-25 17:45:38,835::task::993::Storage.TaskManager.Task::(_decref) Task=`0ef1e51f-51fb-4589-866c-c2c45609e9bd`::ref 0 aborting False >Thread-71051::DEBUG::2014-09-25 17:45:38,835::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71051::DEBUG::2014-09-25 17:45:38,835::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:45:39,424::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:39,425::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:39,425::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:39,425::__init__::232::IOProcess::(_processLogs) (109309) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:39,426::__init__::232::IOProcess::(_processLogs) (109309) Queuing response >Thread-79::DEBUG::2014-09-25 17:45:39,427::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:45:39,443::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00579248 s, 97.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:39,445::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:39,445::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:39,445::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:39,445::__init__::232::IOProcess::(_processLogs) (109310) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:39,451::__init__::232::IOProcess::(_processLogs) (109310) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:39,453::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:39,453::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:39,454::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:39,454::__init__::232::IOProcess::(_processLogs) (109311) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:39,454::__init__::232::IOProcess::(_processLogs) (109311) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:39,455::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:39,456::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:39,456::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:39,456::__init__::232::IOProcess::(_processLogs) (109312) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:39,457::__init__::232::IOProcess::(_processLogs) (109312) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:39,457::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:39,457::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:39,458::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:39,458::__init__::232::IOProcess::(_processLogs) (109313) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:39,458::__init__::232::IOProcess::(_processLogs) (109313) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:39,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:39,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.239129 s, 4.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:41,790::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:41,792::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71052::DEBUG::2014-09-25 17:45:41,794::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:41,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:41,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209453 s, 48.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:43,791::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:43,791::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:43,792::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:43,792::__init__::232::IOProcess::(_processLogs) (109314) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:43,793::__init__::232::IOProcess::(_processLogs) (109314) Queuing response >Thread-81::DEBUG::2014-09-25 17:45:43,794::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:45:43,810::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00353892 s, 105 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:43,812::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:43,812::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:43,812::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:43,813::__init__::232::IOProcess::(_processLogs) (109315) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:43,818::__init__::232::IOProcess::(_processLogs) (109315) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:43,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:43,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159626 s, 64.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:44,809::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:44,811::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71053::DEBUG::2014-09-25 17:45:44,813::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:45,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:46,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0374517 s, 27.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:46,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:46,073::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:46,075::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:46,077::__init__::232::IOProcess::(_processLogs) (109316) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:46,080::__init__::232::IOProcess::(_processLogs) (109316) Queuing response >Thread-80::DEBUG::2014-09-25 17:45:46,082::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:45:46,095::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000487459 s, 775 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:46,099::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:46,101::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:46,103::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:46,105::__init__::232::IOProcess::(_processLogs) (109317) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:46,108::__init__::232::IOProcess::(_processLogs) (109317) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:47,829::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:47,831::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71054::DEBUG::2014-09-25 17:45:47,833::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:48,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:48,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209949 s, 48.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:48,865::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71055::DEBUG::2014-09-25 17:45:48,866::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:45:48,866::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71055::DEBUG::2014-09-25 17:45:48,869::task::595::Storage.TaskManager.Task::(_updateState) Task=`23453ab2-94e4-4109-a2c2-c9738b8dc822`::moving from state init -> state preparing >Thread-71055::INFO::2014-09-25 17:45:48,869::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71055::INFO::2014-09-25 17:45:48,870::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71055::DEBUG::2014-09-25 17:45:48,870::task::1191::Storage.TaskManager.Task::(prepare) Task=`23453ab2-94e4-4109-a2c2-c9738b8dc822`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71055::DEBUG::2014-09-25 17:45:48,870::task::595::Storage.TaskManager.Task::(_updateState) Task=`23453ab2-94e4-4109-a2c2-c9738b8dc822`::moving from state preparing -> state finished >Thread-71055::DEBUG::2014-09-25 17:45:48,870::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71055::DEBUG::2014-09-25 17:45:48,870::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71055::DEBUG::2014-09-25 17:45:48,871::task::993::Storage.TaskManager.Task::(_decref) Task=`23453ab2-94e4-4109-a2c2-c9738b8dc822`::ref 0 aborting False >Thread-71055::DEBUG::2014-09-25 17:45:48,871::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71055::DEBUG::2014-09-25 17:45:48,871::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:48,885::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71056::DEBUG::2014-09-25 17:45:48,886::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:45:48,886::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71056::DEBUG::2014-09-25 17:45:48,888::task::595::Storage.TaskManager.Task::(_updateState) Task=`1ef7ed6e-2fb8-48e7-8119-c2c9a6bbb17f`::moving from state init -> state preparing >Thread-71056::INFO::2014-09-25 17:45:48,888::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71056::DEBUG::2014-09-25 17:45:48,889::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`eb6c4b2b-5baf-41dc-be2e-613551fd28b2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71056::DEBUG::2014-09-25 17:45:48,889::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71056::DEBUG::2014-09-25 17:45:48,890::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71056::DEBUG::2014-09-25 17:45:48,890::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`eb6c4b2b-5baf-41dc-be2e-613551fd28b2`::Granted request >Thread-71056::DEBUG::2014-09-25 17:45:48,890::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1ef7ed6e-2fb8-48e7-8119-c2c9a6bbb17f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71056::DEBUG::2014-09-25 17:45:48,891::task::993::Storage.TaskManager.Task::(_decref) Task=`1ef7ed6e-2fb8-48e7-8119-c2c9a6bbb17f`::ref 1 aborting False >Thread-71056::INFO::2014-09-25 17:45:48,895::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71056::DEBUG::2014-09-25 17:45:48,895::task::1191::Storage.TaskManager.Task::(prepare) Task=`1ef7ed6e-2fb8-48e7-8119-c2c9a6bbb17f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71056::DEBUG::2014-09-25 17:45:48,895::task::595::Storage.TaskManager.Task::(_updateState) Task=`1ef7ed6e-2fb8-48e7-8119-c2c9a6bbb17f`::moving from state preparing -> state finished >Thread-71056::DEBUG::2014-09-25 17:45:48,896::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71056::DEBUG::2014-09-25 17:45:48,896::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71056::DEBUG::2014-09-25 17:45:48,896::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71056::DEBUG::2014-09-25 17:45:48,897::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71056::DEBUG::2014-09-25 17:45:48,897::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71056::DEBUG::2014-09-25 17:45:48,897::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71056::DEBUG::2014-09-25 17:45:48,897::task::993::Storage.TaskManager.Task::(_decref) Task=`1ef7ed6e-2fb8-48e7-8119-c2c9a6bbb17f`::ref 0 aborting False >Thread-71056::DEBUG::2014-09-25 17:45:48,898::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71056::DEBUG::2014-09-25 17:45:48,898::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:45:49,460::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:49,461::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:49,461::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:49,461::__init__::232::IOProcess::(_processLogs) (109318) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:49,466::__init__::232::IOProcess::(_processLogs) (109318) Queuing response >Thread-79::DEBUG::2014-09-25 17:45:49,467::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:45:49,486::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00406391 s, 138 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:49,487::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:49,488::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:49,488::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:49,488::__init__::232::IOProcess::(_processLogs) (109319) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:49,489::__init__::232::IOProcess::(_processLogs) (109319) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:49,490::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:49,490::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:49,490::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:49,491::__init__::232::IOProcess::(_processLogs) (109320) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:49,495::__init__::232::IOProcess::(_processLogs) (109320) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:49,496::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:49,496::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:49,496::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:49,497::__init__::232::IOProcess::(_processLogs) (109321) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:49,499::__init__::232::IOProcess::(_processLogs) (109321) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:49,500::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:49,500::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:49,500::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:49,501::__init__::232::IOProcess::(_processLogs) (109322) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:49,504::__init__::232::IOProcess::(_processLogs) (109322) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:50,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:50,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151235 s, 67.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:50,850::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:50,851::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71057::DEBUG::2014-09-25 17:45:50,853::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:52,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:52,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153956 s, 66.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:53,820::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:53,820::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:53,820::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:53,821::__init__::232::IOProcess::(_processLogs) (109323) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:53,823::__init__::232::IOProcess::(_processLogs) (109323) Queuing response >Thread-81::DEBUG::2014-09-25 17:45:53,823::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:45:53,840::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00354902 s, 105 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:53,841::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:53,842::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:53,842::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:53,842::__init__::232::IOProcess::(_processLogs) (109324) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:53,844::__init__::232::IOProcess::(_processLogs) (109324) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:53,868::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:53,869::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71058::DEBUG::2014-09-25 17:45:53,872::task::595::Storage.TaskManager.Task::(_updateState) Task=`234a524a-b424-4f9a-a3ad-3d87050b1b83`::moving from state init -> state preparing >Thread-71058::INFO::2014-09-25 17:45:53,873::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71058::INFO::2014-09-25 17:45:53,873::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00406391', 'lastCheck': '4.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000487459', 'lastCheck': '7.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00354902', 'lastCheck': '0.0', 'valid': True}} >Thread-71058::DEBUG::2014-09-25 17:45:53,873::task::1191::Storage.TaskManager.Task::(prepare) Task=`234a524a-b424-4f9a-a3ad-3d87050b1b83`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00406391', 'lastCheck': '4.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000487459', 'lastCheck': '7.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00354902', 'lastCheck': '0.0', 'valid': True}} >Thread-71058::DEBUG::2014-09-25 17:45:53,874::task::595::Storage.TaskManager.Task::(_updateState) Task=`234a524a-b424-4f9a-a3ad-3d87050b1b83`::moving from state preparing -> state finished >Thread-71058::DEBUG::2014-09-25 17:45:53,874::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71058::DEBUG::2014-09-25 17:45:53,874::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71058::DEBUG::2014-09-25 17:45:53,874::task::993::Storage.TaskManager.Task::(_decref) Task=`234a524a-b424-4f9a-a3ad-3d87050b1b83`::ref 0 aborting False >Thread-71058::DEBUG::2014-09-25 17:45:53,877::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:53,896::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:53,898::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71059::DEBUG::2014-09-25 17:45:53,899::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:54,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:54,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016922 s, 60.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:56,111::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:56,111::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:56,112::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:56,112::__init__::232::IOProcess::(_processLogs) (109325) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:56,112::__init__::232::IOProcess::(_processLogs) (109325) Queuing response >Thread-80::DEBUG::2014-09-25 17:45:56,113::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:45:56,122::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000493848 s, 765 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:56,124::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:56,124::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:56,124::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:56,124::__init__::232::IOProcess::(_processLogs) (109326) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:56,125::__init__::232::IOProcess::(_processLogs) (109326) Queuing response >Dummy-148::DEBUG::2014-09-25 17:45:56,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:56,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015469 s, 66.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:56,964::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:56,965::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71060::DEBUG::2014-09-25 17:45:56,967::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:45:58,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:45:58,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130037 s, 78.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:58,927::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:58,929::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71061::DEBUG::2014-09-25 17:45:58,929::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71061::DEBUG::2014-09-25 17:45:58,931::task::595::Storage.TaskManager.Task::(_updateState) Task=`35455891-0c1d-4673-b221-bed296c0aa3e`::moving from state init -> state preparing >Thread-71061::INFO::2014-09-25 17:45:58,931::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71061::INFO::2014-09-25 17:45:58,932::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71061::DEBUG::2014-09-25 17:45:58,932::task::1191::Storage.TaskManager.Task::(prepare) Task=`35455891-0c1d-4673-b221-bed296c0aa3e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71061::DEBUG::2014-09-25 17:45:58,932::task::595::Storage.TaskManager.Task::(_updateState) Task=`35455891-0c1d-4673-b221-bed296c0aa3e`::moving from state preparing -> state finished >Thread-71061::DEBUG::2014-09-25 17:45:58,932::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71061::DEBUG::2014-09-25 17:45:58,932::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71061::DEBUG::2014-09-25 17:45:58,933::task::993::Storage.TaskManager.Task::(_decref) Task=`35455891-0c1d-4673-b221-bed296c0aa3e`::ref 0 aborting False >Thread-71061::DEBUG::2014-09-25 17:45:58,933::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71061::DEBUG::2014-09-25 17:45:58,933::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:58,948::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:58,950::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71062::DEBUG::2014-09-25 17:45:58,950::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71062::DEBUG::2014-09-25 17:45:58,951::task::595::Storage.TaskManager.Task::(_updateState) Task=`e532a583-3a11-49de-a7ff-f633a8e670eb`::moving from state init -> state preparing >Thread-71062::INFO::2014-09-25 17:45:58,952::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71062::DEBUG::2014-09-25 17:45:58,952::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d305c5e4-d74a-4b7c-927b-947084c4f8b1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71062::DEBUG::2014-09-25 17:45:58,953::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71062::DEBUG::2014-09-25 17:45:58,953::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71062::DEBUG::2014-09-25 17:45:58,953::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d305c5e4-d74a-4b7c-927b-947084c4f8b1`::Granted request >Thread-71062::DEBUG::2014-09-25 17:45:58,954::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e532a583-3a11-49de-a7ff-f633a8e670eb`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71062::DEBUG::2014-09-25 17:45:58,954::task::993::Storage.TaskManager.Task::(_decref) Task=`e532a583-3a11-49de-a7ff-f633a8e670eb`::ref 1 aborting False >Thread-71062::INFO::2014-09-25 17:45:58,959::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71062::DEBUG::2014-09-25 17:45:58,959::task::1191::Storage.TaskManager.Task::(prepare) Task=`e532a583-3a11-49de-a7ff-f633a8e670eb`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71062::DEBUG::2014-09-25 17:45:58,959::task::595::Storage.TaskManager.Task::(_updateState) Task=`e532a583-3a11-49de-a7ff-f633a8e670eb`::moving from state preparing -> state finished >Thread-71062::DEBUG::2014-09-25 17:45:58,959::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71062::DEBUG::2014-09-25 17:45:58,959::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71062::DEBUG::2014-09-25 17:45:58,960::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71062::DEBUG::2014-09-25 17:45:58,960::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71062::DEBUG::2014-09-25 17:45:58,960::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71062::DEBUG::2014-09-25 17:45:58,961::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71062::DEBUG::2014-09-25 17:45:58,961::task::993::Storage.TaskManager.Task::(_decref) Task=`e532a583-3a11-49de-a7ff-f633a8e670eb`::ref 0 aborting False >Thread-71062::DEBUG::2014-09-25 17:45:58,961::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71062::DEBUG::2014-09-25 17:45:58,962::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:45:59,507::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:59,507::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:59,508::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:59,508::__init__::232::IOProcess::(_processLogs) (109327) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:59,510::__init__::232::IOProcess::(_processLogs) (109327) Queuing response >Thread-79::DEBUG::2014-09-25 17:45:59,511::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:45:59,528::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000984853 s, 571 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:45:59,529::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:59,530::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:59,530::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:59,530::__init__::232::IOProcess::(_processLogs) (109328) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:45:59,536::__init__::232::IOProcess::(_processLogs) (109328) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:59,537::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:59,537::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:59,538::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:59,538::__init__::232::IOProcess::(_processLogs) (109329) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:59,538::__init__::232::IOProcess::(_processLogs) (109329) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:59,539::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:59,539::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:59,540::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:59,540::__init__::232::IOProcess::(_processLogs) (109330) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:59,541::__init__::232::IOProcess::(_processLogs) (109330) Queuing response >Thread-29::DEBUG::2014-09-25 17:45:59,541::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:45:59,541::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:45:59,542::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:45:59,543::__init__::232::IOProcess::(_processLogs) (109331) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:45:59,543::__init__::232::IOProcess::(_processLogs) (109331) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:45:59,985::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:45:59,987::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71063::DEBUG::2014-09-25 17:45:59,989::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:00,544::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:00,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173835 s, 58.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:46:02,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:02,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175015 s, 58.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:03,006::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:03,007::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71064::DEBUG::2014-09-25 17:46:03,009::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:46:03,846::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:03,847::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:03,847::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:03,847::__init__::232::IOProcess::(_processLogs) (109332) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:03,862::__init__::232::IOProcess::(_processLogs) (109332) Queuing response >Thread-81::DEBUG::2014-09-25 17:46:03,863::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:46:03,888::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00345598 s, 107 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:03,889::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:03,890::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:03,890::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:03,890::__init__::232::IOProcess::(_processLogs) (109333) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:03,916::__init__::232::IOProcess::(_processLogs) (109333) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:04,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:04,670::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149637 s, 68.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:06,025::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:06,027::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71065::DEBUG::2014-09-25 17:46:06,029::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:46:06,127::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:06,127::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:06,128::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:06,128::__init__::232::IOProcess::(_processLogs) (109334) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:06,128::__init__::232::IOProcess::(_processLogs) (109334) Queuing response >Thread-80::DEBUG::2014-09-25 17:46:06,130::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:46:06,138::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00036494 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:06,139::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:06,140::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:06,140::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:06,140::__init__::232::IOProcess::(_processLogs) (109335) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:06,141::__init__::232::IOProcess::(_processLogs) (109335) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:06,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:06,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148863 s, 68.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:46:08,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:08,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138733 s, 73.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:09,036::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:09,037::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71066::DEBUG::2014-09-25 17:46:09,037::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71066::DEBUG::2014-09-25 17:46:09,039::task::595::Storage.TaskManager.Task::(_updateState) Task=`a01793ad-c150-48b0-a5f0-a3fd835584bc`::moving from state init -> state preparing >Thread-71066::INFO::2014-09-25 17:46:09,039::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71066::INFO::2014-09-25 17:46:09,040::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71066::DEBUG::2014-09-25 17:46:09,040::task::1191::Storage.TaskManager.Task::(prepare) Task=`a01793ad-c150-48b0-a5f0-a3fd835584bc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71066::DEBUG::2014-09-25 17:46:09,040::task::595::Storage.TaskManager.Task::(_updateState) Task=`a01793ad-c150-48b0-a5f0-a3fd835584bc`::moving from state preparing -> state finished >Thread-71066::DEBUG::2014-09-25 17:46:09,041::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71066::DEBUG::2014-09-25 17:46:09,041::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71066::DEBUG::2014-09-25 17:46:09,041::task::993::Storage.TaskManager.Task::(_decref) Task=`a01793ad-c150-48b0-a5f0-a3fd835584bc`::ref 0 aborting False >Thread-71066::DEBUG::2014-09-25 17:46:09,041::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71066::DEBUG::2014-09-25 17:46:09,042::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:09,044::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:09,045::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71067::DEBUG::2014-09-25 17:46:09,047::task::595::Storage.TaskManager.Task::(_updateState) Task=`70e3b74e-2785-4891-884c-add12f04e1a7`::moving from state init -> state preparing >Thread-71067::INFO::2014-09-25 17:46:09,047::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71067::INFO::2014-09-25 17:46:09,048::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000984853', 'lastCheck': '9.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00036494', 'lastCheck': '2.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00345598', 'lastCheck': '5.1', 'valid': True}} >Thread-71067::DEBUG::2014-09-25 17:46:09,048::task::1191::Storage.TaskManager.Task::(prepare) Task=`70e3b74e-2785-4891-884c-add12f04e1a7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000984853', 'lastCheck': '9.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00036494', 'lastCheck': '2.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00345598', 'lastCheck': '5.1', 'valid': True}} >Thread-71067::DEBUG::2014-09-25 17:46:09,048::task::595::Storage.TaskManager.Task::(_updateState) Task=`70e3b74e-2785-4891-884c-add12f04e1a7`::moving from state preparing -> state finished >Thread-71067::DEBUG::2014-09-25 17:46:09,048::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71067::DEBUG::2014-09-25 17:46:09,049::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71067::DEBUG::2014-09-25 17:46:09,049::task::993::Storage.TaskManager.Task::(_decref) Task=`70e3b74e-2785-4891-884c-add12f04e1a7`::ref 0 aborting False >Thread-71067::DEBUG::2014-09-25 17:46:09,051::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:09,056::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:09,058::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71068::DEBUG::2014-09-25 17:46:09,058::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71068::DEBUG::2014-09-25 17:46:09,059::task::595::Storage.TaskManager.Task::(_updateState) Task=`c0078b84-8953-464b-bfbf-c518a8edda0d`::moving from state init -> state preparing >Thread-71068::INFO::2014-09-25 17:46:09,060::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71068::DEBUG::2014-09-25 17:46:09,060::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`627cf549-cef7-4cda-9947-702c8fef6114`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71068::DEBUG::2014-09-25 17:46:09,061::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71068::DEBUG::2014-09-25 17:46:09,061::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71068::DEBUG::2014-09-25 17:46:09,061::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`627cf549-cef7-4cda-9947-702c8fef6114`::Granted request >Thread-71068::DEBUG::2014-09-25 17:46:09,062::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c0078b84-8953-464b-bfbf-c518a8edda0d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71068::DEBUG::2014-09-25 17:46:09,062::task::993::Storage.TaskManager.Task::(_decref) Task=`c0078b84-8953-464b-bfbf-c518a8edda0d`::ref 1 aborting False >Thread-71068::INFO::2014-09-25 17:46:09,066::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71068::DEBUG::2014-09-25 17:46:09,067::task::1191::Storage.TaskManager.Task::(prepare) Task=`c0078b84-8953-464b-bfbf-c518a8edda0d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71068::DEBUG::2014-09-25 17:46:09,067::task::595::Storage.TaskManager.Task::(_updateState) Task=`c0078b84-8953-464b-bfbf-c518a8edda0d`::moving from state preparing -> state finished >Thread-71068::DEBUG::2014-09-25 17:46:09,067::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71068::DEBUG::2014-09-25 17:46:09,067::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71068::DEBUG::2014-09-25 17:46:09,068::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71068::DEBUG::2014-09-25 17:46:09,068::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71068::DEBUG::2014-09-25 17:46:09,068::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71068::DEBUG::2014-09-25 17:46:09,069::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71068::DEBUG::2014-09-25 17:46:09,069::task::993::Storage.TaskManager.Task::(_decref) Task=`c0078b84-8953-464b-bfbf-c518a8edda0d`::ref 0 aborting False >Thread-71068::DEBUG::2014-09-25 17:46:09,069::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71068::DEBUG::2014-09-25 17:46:09,070::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:09,076::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:09,077::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71069::DEBUG::2014-09-25 17:46:09,079::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:46:09,544::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:09,544::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:09,544::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:09,544::__init__::232::IOProcess::(_processLogs) (109336) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:09,545::__init__::232::IOProcess::(_processLogs) (109336) Queuing response >Thread-79::DEBUG::2014-09-25 17:46:09,546::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:46:09,555::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000360113 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:09,556::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:09,556::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:09,557::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:09,557::__init__::232::IOProcess::(_processLogs) (109337) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:09,559::__init__::232::IOProcess::(_processLogs) (109337) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:09,565::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:09,567::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:09,567::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:09,567::__init__::232::IOProcess::(_processLogs) (109338) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:09,568::__init__::232::IOProcess::(_processLogs) (109338) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:09,568::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:09,569::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:09,569::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:09,570::__init__::232::IOProcess::(_processLogs) (109339) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:09,570::__init__::232::IOProcess::(_processLogs) (109339) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:09,571::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:09,571::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:09,572::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:09,572::__init__::232::IOProcess::(_processLogs) (109340) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:09,573::__init__::232::IOProcess::(_processLogs) (109340) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:10,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:10,801::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135464 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:12,151::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:12,153::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71070::DEBUG::2014-09-25 17:46:12,155::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:12,821::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:12,843::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135741 s, 75.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:13,919::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:13,919::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:13,920::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:13,920::__init__::232::IOProcess::(_processLogs) (109341) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:13,925::__init__::232::IOProcess::(_processLogs) (109341) Queuing response >Thread-81::DEBUG::2014-09-25 17:46:13,926::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:46:13,944::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00799578 s, 46.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:13,945::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:13,945::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:13,946::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:13,946::__init__::232::IOProcess::(_processLogs) (109342) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:13,951::__init__::232::IOProcess::(_processLogs) (109342) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:14,862::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:14,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163625 s, 62.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:15,171::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:15,173::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71071::DEBUG::2014-09-25 17:46:15,175::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:46:16,144::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:16,145::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:16,146::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:16,148::__init__::232::IOProcess::(_processLogs) (109343) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:16,149::__init__::232::IOProcess::(_processLogs) (109343) Queuing response >Thread-80::DEBUG::2014-09-25 17:46:16,151::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:46:16,161::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000405066 s, 933 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:16,164::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:16,165::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:16,166::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:16,167::__init__::232::IOProcess::(_processLogs) (109344) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:16,169::__init__::232::IOProcess::(_processLogs) (109344) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:16,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:16,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0236272 s, 43.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:18,190::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:18,191::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71072::DEBUG::2014-09-25 17:46:18,193::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:19,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:19,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138478 s, 73.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:19,123::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71073::DEBUG::2014-09-25 17:46:19,125::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:46:19,125::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71073::DEBUG::2014-09-25 17:46:19,128::task::595::Storage.TaskManager.Task::(_updateState) Task=`48a1896a-3c9f-4cb8-b38e-2f27b1914a01`::moving from state init -> state preparing >Thread-71073::INFO::2014-09-25 17:46:19,128::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71073::INFO::2014-09-25 17:46:19,129::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71073::DEBUG::2014-09-25 17:46:19,129::task::1191::Storage.TaskManager.Task::(prepare) Task=`48a1896a-3c9f-4cb8-b38e-2f27b1914a01`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71073::DEBUG::2014-09-25 17:46:19,129::task::595::Storage.TaskManager.Task::(_updateState) Task=`48a1896a-3c9f-4cb8-b38e-2f27b1914a01`::moving from state preparing -> state finished >Thread-71073::DEBUG::2014-09-25 17:46:19,129::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71073::DEBUG::2014-09-25 17:46:19,130::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71073::DEBUG::2014-09-25 17:46:19,130::task::993::Storage.TaskManager.Task::(_decref) Task=`48a1896a-3c9f-4cb8-b38e-2f27b1914a01`::ref 0 aborting False >Thread-71073::DEBUG::2014-09-25 17:46:19,130::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71073::DEBUG::2014-09-25 17:46:19,130::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:19,144::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:19,145::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71074::DEBUG::2014-09-25 17:46:19,145::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71074::DEBUG::2014-09-25 17:46:19,147::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b15d283-72bd-4897-b15b-2eb962ebec37`::moving from state init -> state preparing >Thread-71074::INFO::2014-09-25 17:46:19,147::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71074::DEBUG::2014-09-25 17:46:19,148::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b3fa00c3-64ce-411a-9fa6-f4e9c9df4cde`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71074::DEBUG::2014-09-25 17:46:19,148::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71074::DEBUG::2014-09-25 17:46:19,148::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71074::DEBUG::2014-09-25 17:46:19,149::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b3fa00c3-64ce-411a-9fa6-f4e9c9df4cde`::Granted request >Thread-71074::DEBUG::2014-09-25 17:46:19,150::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1b15d283-72bd-4897-b15b-2eb962ebec37`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71074::DEBUG::2014-09-25 17:46:19,150::task::993::Storage.TaskManager.Task::(_decref) Task=`1b15d283-72bd-4897-b15b-2eb962ebec37`::ref 1 aborting False >Thread-71074::INFO::2014-09-25 17:46:19,155::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71074::DEBUG::2014-09-25 17:46:19,155::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b15d283-72bd-4897-b15b-2eb962ebec37`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71074::DEBUG::2014-09-25 17:46:19,155::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b15d283-72bd-4897-b15b-2eb962ebec37`::moving from state preparing -> state finished >Thread-71074::DEBUG::2014-09-25 17:46:19,156::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71074::DEBUG::2014-09-25 17:46:19,156::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71074::DEBUG::2014-09-25 17:46:19,156::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71074::DEBUG::2014-09-25 17:46:19,157::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71074::DEBUG::2014-09-25 17:46:19,157::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71074::DEBUG::2014-09-25 17:46:19,157::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71074::DEBUG::2014-09-25 17:46:19,157::task::993::Storage.TaskManager.Task::(_decref) Task=`1b15d283-72bd-4897-b15b-2eb962ebec37`::ref 0 aborting False >Thread-71074::DEBUG::2014-09-25 17:46:19,158::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71074::DEBUG::2014-09-25 17:46:19,158::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:46:19,573::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:19,573::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:19,573::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:19,573::__init__::232::IOProcess::(_processLogs) (109345) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:19,574::__init__::232::IOProcess::(_processLogs) (109345) Queuing response >Thread-79::DEBUG::2014-09-25 17:46:19,575::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:46:19,584::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000402091 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:19,585::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:19,585::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:19,586::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:19,586::__init__::232::IOProcess::(_processLogs) (109346) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:19,586::__init__::232::IOProcess::(_processLogs) (109346) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:19,588::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:19,588::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:19,588::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:19,589::__init__::232::IOProcess::(_processLogs) (109347) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:19,589::__init__::232::IOProcess::(_processLogs) (109347) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:19,590::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:19,590::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:19,590::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:19,591::__init__::232::IOProcess::(_processLogs) (109348) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:19,591::__init__::232::IOProcess::(_processLogs) (109348) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:19,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:19,592::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:19,592::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:19,592::__init__::232::IOProcess::(_processLogs) (109349) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:19,593::__init__::232::IOProcess::(_processLogs) (109349) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:21,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:21,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137104 s, 74.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:21,208::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:21,210::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71075::DEBUG::2014-09-25 17:46:21,212::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:23,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:23,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136533 s, 75.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:23,954::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:23,954::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:23,954::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:23,954::__init__::232::IOProcess::(_processLogs) (109350) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:23,955::__init__::232::IOProcess::(_processLogs) (109350) Queuing response >Thread-81::DEBUG::2014-09-25 17:46:23,956::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:46:23,965::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000498244 s, 745 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:23,967::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:23,967::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:23,967::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:23,967::__init__::232::IOProcess::(_processLogs) (109351) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:23,968::__init__::232::IOProcess::(_processLogs) (109351) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:24,225::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:24,227::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71076::DEBUG::2014-09-25 17:46:24,229::task::595::Storage.TaskManager.Task::(_updateState) Task=`ec53cac2-98c4-4ba2-8c38-2ea2bb6f5aac`::moving from state init -> state preparing >Thread-71076::INFO::2014-09-25 17:46:24,230::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71076::INFO::2014-09-25 17:46:24,230::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000402091', 'lastCheck': '4.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000405066', 'lastCheck': '8.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000498244', 'lastCheck': '0.3', 'valid': True}} >Thread-71076::DEBUG::2014-09-25 17:46:24,230::task::1191::Storage.TaskManager.Task::(prepare) Task=`ec53cac2-98c4-4ba2-8c38-2ea2bb6f5aac`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000402091', 'lastCheck': '4.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000405066', 'lastCheck': '8.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000498244', 'lastCheck': '0.3', 'valid': True}} >Thread-71076::DEBUG::2014-09-25 17:46:24,231::task::595::Storage.TaskManager.Task::(_updateState) Task=`ec53cac2-98c4-4ba2-8c38-2ea2bb6f5aac`::moving from state preparing -> state finished >Thread-71076::DEBUG::2014-09-25 17:46:24,231::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71076::DEBUG::2014-09-25 17:46:24,231::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71076::DEBUG::2014-09-25 17:46:24,231::task::993::Storage.TaskManager.Task::(_decref) Task=`ec53cac2-98c4-4ba2-8c38-2ea2bb6f5aac`::ref 0 aborting False >Thread-71076::DEBUG::2014-09-25 17:46:24,234::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:24,255::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:24,257::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71077::DEBUG::2014-09-25 17:46:24,259::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:25,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:25,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013205 s, 77.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:26,172::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:26,173::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:26,174::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:26,175::__init__::232::IOProcess::(_processLogs) (109352) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:26,177::__init__::232::IOProcess::(_processLogs) (109352) Queuing response >Thread-80::DEBUG::2014-09-25 17:46:26,178::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:46:26,189::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000403855 s, 936 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:26,190::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:26,190::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:26,191::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:26,191::__init__::232::IOProcess::(_processLogs) (109353) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:26,192::__init__::232::IOProcess::(_processLogs) (109353) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:27,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:27,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133452 s, 76.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:27,321::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:27,322::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71078::DEBUG::2014-09-25 17:46:27,324::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:29,190::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:29,191::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71079::DEBUG::2014-09-25 17:46:29,191::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71079::DEBUG::2014-09-25 17:46:29,192::task::595::Storage.TaskManager.Task::(_updateState) Task=`0d40f06a-9af8-4855-b7f2-bcde353f5151`::moving from state init -> state preparing >Thread-71079::INFO::2014-09-25 17:46:29,193::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71079::INFO::2014-09-25 17:46:29,193::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71079::DEBUG::2014-09-25 17:46:29,194::task::1191::Storage.TaskManager.Task::(prepare) Task=`0d40f06a-9af8-4855-b7f2-bcde353f5151`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71079::DEBUG::2014-09-25 17:46:29,194::task::595::Storage.TaskManager.Task::(_updateState) Task=`0d40f06a-9af8-4855-b7f2-bcde353f5151`::moving from state preparing -> state finished >Thread-71079::DEBUG::2014-09-25 17:46:29,194::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71079::DEBUG::2014-09-25 17:46:29,194::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71079::DEBUG::2014-09-25 17:46:29,194::task::993::Storage.TaskManager.Task::(_decref) Task=`0d40f06a-9af8-4855-b7f2-bcde353f5151`::ref 0 aborting False >Thread-71079::DEBUG::2014-09-25 17:46:29,195::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71079::DEBUG::2014-09-25 17:46:29,195::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:29,209::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:29,210::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71080::DEBUG::2014-09-25 17:46:29,210::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71080::DEBUG::2014-09-25 17:46:29,212::task::595::Storage.TaskManager.Task::(_updateState) Task=`34d1c537-f445-4ffd-9b80-f04ae44d8b8a`::moving from state init -> state preparing >Thread-71080::INFO::2014-09-25 17:46:29,212::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71080::DEBUG::2014-09-25 17:46:29,213::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`29d241b6-ff2d-4b45-b1dd-2efd9d78dfb1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71080::DEBUG::2014-09-25 17:46:29,213::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71080::DEBUG::2014-09-25 17:46:29,214::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71080::DEBUG::2014-09-25 17:46:29,214::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`29d241b6-ff2d-4b45-b1dd-2efd9d78dfb1`::Granted request >Thread-71080::DEBUG::2014-09-25 17:46:29,214::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`34d1c537-f445-4ffd-9b80-f04ae44d8b8a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71080::DEBUG::2014-09-25 17:46:29,215::task::993::Storage.TaskManager.Task::(_decref) Task=`34d1c537-f445-4ffd-9b80-f04ae44d8b8a`::ref 1 aborting False >Thread-71080::INFO::2014-09-25 17:46:29,219::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71080::DEBUG::2014-09-25 17:46:29,219::task::1191::Storage.TaskManager.Task::(prepare) Task=`34d1c537-f445-4ffd-9b80-f04ae44d8b8a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71080::DEBUG::2014-09-25 17:46:29,220::task::595::Storage.TaskManager.Task::(_updateState) Task=`34d1c537-f445-4ffd-9b80-f04ae44d8b8a`::moving from state preparing -> state finished >Thread-71080::DEBUG::2014-09-25 17:46:29,220::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71080::DEBUG::2014-09-25 17:46:29,220::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71080::DEBUG::2014-09-25 17:46:29,220::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71080::DEBUG::2014-09-25 17:46:29,221::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71080::DEBUG::2014-09-25 17:46:29,221::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71080::DEBUG::2014-09-25 17:46:29,221::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71080::DEBUG::2014-09-25 17:46:29,222::task::993::Storage.TaskManager.Task::(_decref) Task=`34d1c537-f445-4ffd-9b80-f04ae44d8b8a`::ref 0 aborting False >Thread-71080::DEBUG::2014-09-25 17:46:29,222::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71080::DEBUG::2014-09-25 17:46:29,223::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:29,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:29,266::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193321 s, 53.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:29,594::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:29,595::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:29,595::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:29,595::__init__::232::IOProcess::(_processLogs) (109354) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:29,596::__init__::232::IOProcess::(_processLogs) (109354) Queuing response >Thread-79::DEBUG::2014-09-25 17:46:29,597::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:46:29,605::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000443312 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:29,607::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:29,607::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:29,607::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:29,608::__init__::232::IOProcess::(_processLogs) (109355) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:29,608::__init__::232::IOProcess::(_processLogs) (109355) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:29,610::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:29,610::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:29,610::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:29,610::__init__::232::IOProcess::(_processLogs) (109356) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:29,611::__init__::232::IOProcess::(_processLogs) (109356) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:29,612::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:29,612::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:29,612::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:29,612::__init__::232::IOProcess::(_processLogs) (109357) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:29,613::__init__::232::IOProcess::(_processLogs) (109357) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:29,613::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:29,614::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:29,614::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:29,614::__init__::232::IOProcess::(_processLogs) (109358) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:29,614::__init__::232::IOProcess::(_processLogs) (109358) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:30,340::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:30,342::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71081::DEBUG::2014-09-25 17:46:30,344::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:31,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:31,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0195762 s, 52.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:46:33,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:33,360::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:33,364::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71082::DEBUG::2014-09-25 17:46:33,369::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:33,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0219788 s, 46.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:33,970::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:33,971::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:33,971::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:33,971::__init__::232::IOProcess::(_processLogs) (109359) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:33,972::__init__::232::IOProcess::(_processLogs) (109359) Queuing response >Thread-81::DEBUG::2014-09-25 17:46:33,973::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:46:33,982::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000579705 s, 640 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:33,983::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:33,983::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:33,984::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:33,984::__init__::232::IOProcess::(_processLogs) (109360) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:33,984::__init__::232::IOProcess::(_processLogs) (109360) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:35,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:35,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013228 s, 77.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:36,194::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:36,194::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:36,194::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:36,195::__init__::232::IOProcess::(_processLogs) (109361) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:36,195::__init__::232::IOProcess::(_processLogs) (109361) Queuing response >Thread-80::DEBUG::2014-09-25 17:46:36,196::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:46:36,205::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000484833 s, 780 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:36,206::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:36,207::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:36,207::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:36,207::__init__::232::IOProcess::(_processLogs) (109362) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:36,208::__init__::232::IOProcess::(_processLogs) (109362) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:36,386::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:36,388::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71083::DEBUG::2014-09-25 17:46:36,390::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:37,435::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:37,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133449 s, 76.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:39,253::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71084::DEBUG::2014-09-25 17:46:39,254::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:46:39,254::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71084::DEBUG::2014-09-25 17:46:39,256::task::595::Storage.TaskManager.Task::(_updateState) Task=`36a970f7-33f0-40fa-8e33-18568defd57b`::moving from state init -> state preparing >Thread-71084::INFO::2014-09-25 17:46:39,257::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71084::INFO::2014-09-25 17:46:39,257::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71084::DEBUG::2014-09-25 17:46:39,257::task::1191::Storage.TaskManager.Task::(prepare) Task=`36a970f7-33f0-40fa-8e33-18568defd57b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71084::DEBUG::2014-09-25 17:46:39,258::task::595::Storage.TaskManager.Task::(_updateState) Task=`36a970f7-33f0-40fa-8e33-18568defd57b`::moving from state preparing -> state finished >Thread-71084::DEBUG::2014-09-25 17:46:39,258::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71084::DEBUG::2014-09-25 17:46:39,258::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71084::DEBUG::2014-09-25 17:46:39,258::task::993::Storage.TaskManager.Task::(_decref) Task=`36a970f7-33f0-40fa-8e33-18568defd57b`::ref 0 aborting False >Thread-71084::DEBUG::2014-09-25 17:46:39,259::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71084::DEBUG::2014-09-25 17:46:39,259::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:39,273::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71085::DEBUG::2014-09-25 17:46:39,274::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:46:39,274::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71085::DEBUG::2014-09-25 17:46:39,276::task::595::Storage.TaskManager.Task::(_updateState) Task=`47a1c10c-05c9-4db4-b90e-c96a19640ebb`::moving from state init -> state preparing >Thread-71085::INFO::2014-09-25 17:46:39,276::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71085::DEBUG::2014-09-25 17:46:39,277::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a75ffda8-77fc-424f-9b02-d8b57cf8b8fc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71085::DEBUG::2014-09-25 17:46:39,277::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71085::DEBUG::2014-09-25 17:46:39,277::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71085::DEBUG::2014-09-25 17:46:39,278::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a75ffda8-77fc-424f-9b02-d8b57cf8b8fc`::Granted request >Thread-71085::DEBUG::2014-09-25 17:46:39,278::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`47a1c10c-05c9-4db4-b90e-c96a19640ebb`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71085::DEBUG::2014-09-25 17:46:39,279::task::993::Storage.TaskManager.Task::(_decref) Task=`47a1c10c-05c9-4db4-b90e-c96a19640ebb`::ref 1 aborting False >Thread-71085::INFO::2014-09-25 17:46:39,283::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71085::DEBUG::2014-09-25 17:46:39,283::task::1191::Storage.TaskManager.Task::(prepare) Task=`47a1c10c-05c9-4db4-b90e-c96a19640ebb`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71085::DEBUG::2014-09-25 17:46:39,283::task::595::Storage.TaskManager.Task::(_updateState) Task=`47a1c10c-05c9-4db4-b90e-c96a19640ebb`::moving from state preparing -> state finished >Thread-71085::DEBUG::2014-09-25 17:46:39,284::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71085::DEBUG::2014-09-25 17:46:39,284::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71085::DEBUG::2014-09-25 17:46:39,284::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71085::DEBUG::2014-09-25 17:46:39,285::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71085::DEBUG::2014-09-25 17:46:39,285::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71085::DEBUG::2014-09-25 17:46:39,285::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71085::DEBUG::2014-09-25 17:46:39,285::task::993::Storage.TaskManager.Task::(_decref) Task=`47a1c10c-05c9-4db4-b90e-c96a19640ebb`::ref 0 aborting False >Thread-71085::DEBUG::2014-09-25 17:46:39,286::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71085::DEBUG::2014-09-25 17:46:39,286::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:39,404::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:39,405::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71086::DEBUG::2014-09-25 17:46:39,408::task::595::Storage.TaskManager.Task::(_updateState) Task=`32bfee8e-1d51-4101-bf0d-9b4b5406d32d`::moving from state init -> state preparing >Thread-71086::INFO::2014-09-25 17:46:39,408::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71086::INFO::2014-09-25 17:46:39,408::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443312', 'lastCheck': '9.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000484833', 'lastCheck': '3.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000579705', 'lastCheck': '5.4', 'valid': True}} >Thread-71086::DEBUG::2014-09-25 17:46:39,409::task::1191::Storage.TaskManager.Task::(prepare) Task=`32bfee8e-1d51-4101-bf0d-9b4b5406d32d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443312', 'lastCheck': '9.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000484833', 'lastCheck': '3.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000579705', 'lastCheck': '5.4', 'valid': True}} >Thread-71086::DEBUG::2014-09-25 17:46:39,409::task::595::Storage.TaskManager.Task::(_updateState) Task=`32bfee8e-1d51-4101-bf0d-9b4b5406d32d`::moving from state preparing -> state finished >Thread-71086::DEBUG::2014-09-25 17:46:39,409::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71086::DEBUG::2014-09-25 17:46:39,409::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71086::DEBUG::2014-09-25 17:46:39,409::task::993::Storage.TaskManager.Task::(_decref) Task=`32bfee8e-1d51-4101-bf0d-9b4b5406d32d`::ref 0 aborting False >Thread-71086::DEBUG::2014-09-25 17:46:39,412::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:39,431::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:39,432::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71087::DEBUG::2014-09-25 17:46:39,434::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:39,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:39,499::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130653 s, 78.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:39,616::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:39,616::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:39,616::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:39,617::__init__::232::IOProcess::(_processLogs) (109363) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:39,617::__init__::232::IOProcess::(_processLogs) (109363) Queuing response >Thread-79::DEBUG::2014-09-25 17:46:39,618::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:46:39,627::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000610663 s, 920 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:39,628::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:39,629::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:39,629::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:39,629::__init__::232::IOProcess::(_processLogs) (109364) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:39,630::__init__::232::IOProcess::(_processLogs) (109364) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:39,631::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:39,632::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:39,632::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:39,632::__init__::232::IOProcess::(_processLogs) (109365) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:39,633::__init__::232::IOProcess::(_processLogs) (109365) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:39,633::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:39,633::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:39,634::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:39,634::__init__::232::IOProcess::(_processLogs) (109366) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:39,634::__init__::232::IOProcess::(_processLogs) (109366) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:39,635::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:39,635::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:39,636::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:39,637::__init__::232::IOProcess::(_processLogs) (109367) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:39,637::__init__::232::IOProcess::(_processLogs) (109367) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:41,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:41,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137313 s, 74.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:42,498::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:42,500::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71088::DEBUG::2014-09-25 17:46:42,502::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:43,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:43,603::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174834 s, 58.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:43,987::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:43,987::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:43,987::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:43,988::__init__::232::IOProcess::(_processLogs) (109368) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:43,988::__init__::232::IOProcess::(_processLogs) (109368) Queuing response >Thread-81::DEBUG::2014-09-25 17:46:43,989::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:46:44,003::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00210011 s, 177 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:44,004::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:44,004::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:44,005::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:44,005::__init__::232::IOProcess::(_processLogs) (109369) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:44,007::__init__::232::IOProcess::(_processLogs) (109369) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:45,518::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:45,520::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71089::DEBUG::2014-09-25 17:46:45,523::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:45,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:45,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153319 s, 66.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:46,211::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:46,213::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:46,214::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:46,215::__init__::232::IOProcess::(_processLogs) (109370) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:46,217::__init__::232::IOProcess::(_processLogs) (109370) Queuing response >Thread-80::DEBUG::2014-09-25 17:46:46,219::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:46:46,230::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000387855 s, 975 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:46,233::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:46,234::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:46,235::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:46,237::__init__::232::IOProcess::(_processLogs) (109371) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:46,239::__init__::232::IOProcess::(_processLogs) (109371) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:47,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:47,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133464 s, 76.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:48,539::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:48,541::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71090::DEBUG::2014-09-25 17:46:48,543::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:49,315::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71091::DEBUG::2014-09-25 17:46:49,316::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:46:49,316::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71091::DEBUG::2014-09-25 17:46:49,318::task::595::Storage.TaskManager.Task::(_updateState) Task=`c306ef8a-fc5d-416e-85d6-a8be7e3bd07d`::moving from state init -> state preparing >Thread-71091::INFO::2014-09-25 17:46:49,319::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71091::INFO::2014-09-25 17:46:49,319::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71091::DEBUG::2014-09-25 17:46:49,319::task::1191::Storage.TaskManager.Task::(prepare) Task=`c306ef8a-fc5d-416e-85d6-a8be7e3bd07d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71091::DEBUG::2014-09-25 17:46:49,320::task::595::Storage.TaskManager.Task::(_updateState) Task=`c306ef8a-fc5d-416e-85d6-a8be7e3bd07d`::moving from state preparing -> state finished >Thread-71091::DEBUG::2014-09-25 17:46:49,320::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71091::DEBUG::2014-09-25 17:46:49,320::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71091::DEBUG::2014-09-25 17:46:49,320::task::993::Storage.TaskManager.Task::(_decref) Task=`c306ef8a-fc5d-416e-85d6-a8be7e3bd07d`::ref 0 aborting False >Thread-71091::DEBUG::2014-09-25 17:46:49,321::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71091::DEBUG::2014-09-25 17:46:49,321::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:49,334::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71092::DEBUG::2014-09-25 17:46:49,336::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:46:49,336::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71092::DEBUG::2014-09-25 17:46:49,338::task::595::Storage.TaskManager.Task::(_updateState) Task=`67271c74-b02f-4dd7-8630-bf6e0d144e20`::moving from state init -> state preparing >Thread-71092::INFO::2014-09-25 17:46:49,338::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71092::DEBUG::2014-09-25 17:46:49,339::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3177055e-088b-4a45-96ef-177ca3a59365`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71092::DEBUG::2014-09-25 17:46:49,339::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71092::DEBUG::2014-09-25 17:46:49,339::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71092::DEBUG::2014-09-25 17:46:49,340::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3177055e-088b-4a45-96ef-177ca3a59365`::Granted request >Thread-71092::DEBUG::2014-09-25 17:46:49,340::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`67271c74-b02f-4dd7-8630-bf6e0d144e20`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71092::DEBUG::2014-09-25 17:46:49,340::task::993::Storage.TaskManager.Task::(_decref) Task=`67271c74-b02f-4dd7-8630-bf6e0d144e20`::ref 1 aborting False >Thread-71092::INFO::2014-09-25 17:46:49,345::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71092::DEBUG::2014-09-25 17:46:49,345::task::1191::Storage.TaskManager.Task::(prepare) Task=`67271c74-b02f-4dd7-8630-bf6e0d144e20`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71092::DEBUG::2014-09-25 17:46:49,345::task::595::Storage.TaskManager.Task::(_updateState) Task=`67271c74-b02f-4dd7-8630-bf6e0d144e20`::moving from state preparing -> state finished >Thread-71092::DEBUG::2014-09-25 17:46:49,346::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71092::DEBUG::2014-09-25 17:46:49,346::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71092::DEBUG::2014-09-25 17:46:49,346::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71092::DEBUG::2014-09-25 17:46:49,346::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71092::DEBUG::2014-09-25 17:46:49,347::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71092::DEBUG::2014-09-25 17:46:49,347::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71092::DEBUG::2014-09-25 17:46:49,347::task::993::Storage.TaskManager.Task::(_decref) Task=`67271c74-b02f-4dd7-8630-bf6e0d144e20`::ref 0 aborting False >Thread-71092::DEBUG::2014-09-25 17:46:49,348::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71092::DEBUG::2014-09-25 17:46:49,348::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:46:49,637::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:49,637::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:49,638::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:49,638::__init__::232::IOProcess::(_processLogs) (109372) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:49,638::__init__::232::IOProcess::(_processLogs) (109372) Queuing response >Thread-79::DEBUG::2014-09-25 17:46:49,639::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:46:49,648::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000515815 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:49,649::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:49,650::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:49,650::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:49,650::__init__::232::IOProcess::(_processLogs) (109373) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:49,651::__init__::232::IOProcess::(_processLogs) (109373) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:49,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:49,653::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:49,653::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:49,653::__init__::232::IOProcess::(_processLogs) (109374) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:49,653::__init__::232::IOProcess::(_processLogs) (109374) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:49,654::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:49,655::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:49,655::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:49,655::__init__::232::IOProcess::(_processLogs) (109375) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:49,656::__init__::232::IOProcess::(_processLogs) (109375) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:49,657::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:49,657::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:49,657::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:49,657::__init__::232::IOProcess::(_processLogs) (109376) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:49,658::__init__::232::IOProcess::(_processLogs) (109376) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:49,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:49,736::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132967 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:51,561::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:51,563::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71093::DEBUG::2014-09-25 17:46:51,565::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:51,756::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:51,778::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135714 s, 75.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:46:53,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:53,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133766 s, 76.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:54,009::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:54,009::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:54,009::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:54,010::__init__::232::IOProcess::(_processLogs) (109377) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:54,010::__init__::232::IOProcess::(_processLogs) (109377) Queuing response >Thread-81::DEBUG::2014-09-25 17:46:54,011::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:46:54,020::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000393116 s, 944 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:54,021::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:54,022::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:54,022::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:54,022::__init__::232::IOProcess::(_processLogs) (109378) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:54,023::__init__::232::IOProcess::(_processLogs) (109378) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:54,581::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:54,583::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71094::DEBUG::2014-09-25 17:46:54,585::task::595::Storage.TaskManager.Task::(_updateState) Task=`6b412f64-75f5-4507-b7ed-3a6f777811ed`::moving from state init -> state preparing >Thread-71094::INFO::2014-09-25 17:46:54,586::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71094::INFO::2014-09-25 17:46:54,586::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000515815', 'lastCheck': '4.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000387855', 'lastCheck': '8.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000393116', 'lastCheck': '0.6', 'valid': True}} >Thread-71094::DEBUG::2014-09-25 17:46:54,586::task::1191::Storage.TaskManager.Task::(prepare) Task=`6b412f64-75f5-4507-b7ed-3a6f777811ed`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000515815', 'lastCheck': '4.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000387855', 'lastCheck': '8.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000393116', 'lastCheck': '0.6', 'valid': True}} >Thread-71094::DEBUG::2014-09-25 17:46:54,587::task::595::Storage.TaskManager.Task::(_updateState) Task=`6b412f64-75f5-4507-b7ed-3a6f777811ed`::moving from state preparing -> state finished >Thread-71094::DEBUG::2014-09-25 17:46:54,587::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71094::DEBUG::2014-09-25 17:46:54,587::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71094::DEBUG::2014-09-25 17:46:54,587::task::993::Storage.TaskManager.Task::(_decref) Task=`6b412f64-75f5-4507-b7ed-3a6f777811ed`::ref 0 aborting False >Thread-71094::DEBUG::2014-09-25 17:46:54,590::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:54,610::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:54,611::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71095::DEBUG::2014-09-25 17:46:54,613::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:55,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:55,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134038 s, 76.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:56,242::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:56,242::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:56,243::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:46:56,244::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:46:56,244::__init__::232::IOProcess::(_processLogs) (109379) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:56,255::__init__::232::IOProcess::(_processLogs) (109379) Queuing response >Thread-80::DEBUG::2014-09-25 17:46:56,257::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000378334 s, 999 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:56,258::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:56,258::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:56,259::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:56,260::__init__::232::IOProcess::(_processLogs) (109380) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:56,260::__init__::232::IOProcess::(_processLogs) (109380) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:57,677::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:57,678::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71096::DEBUG::2014-09-25 17:46:57,680::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:46:57,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:57,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192572 s, 53.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:59,379::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71097::DEBUG::2014-09-25 17:46:59,380::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:46:59,380::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71097::DEBUG::2014-09-25 17:46:59,383::task::595::Storage.TaskManager.Task::(_updateState) Task=`04e9c4a9-7630-40a0-8265-128208402587`::moving from state init -> state preparing >Thread-71097::INFO::2014-09-25 17:46:59,383::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71097::INFO::2014-09-25 17:46:59,383::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71097::DEBUG::2014-09-25 17:46:59,384::task::1191::Storage.TaskManager.Task::(prepare) Task=`04e9c4a9-7630-40a0-8265-128208402587`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71097::DEBUG::2014-09-25 17:46:59,384::task::595::Storage.TaskManager.Task::(_updateState) Task=`04e9c4a9-7630-40a0-8265-128208402587`::moving from state preparing -> state finished >Thread-71097::DEBUG::2014-09-25 17:46:59,384::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71097::DEBUG::2014-09-25 17:46:59,384::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71097::DEBUG::2014-09-25 17:46:59,385::task::993::Storage.TaskManager.Task::(_decref) Task=`04e9c4a9-7630-40a0-8265-128208402587`::ref 0 aborting False >Thread-71097::DEBUG::2014-09-25 17:46:59,385::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71097::DEBUG::2014-09-25 17:46:59,385::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:46:59,399::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:46:59,400::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71098::DEBUG::2014-09-25 17:46:59,400::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71098::DEBUG::2014-09-25 17:46:59,402::task::595::Storage.TaskManager.Task::(_updateState) Task=`1bdd6480-37c2-4b57-bc20-bdffd5cc71f4`::moving from state init -> state preparing >Thread-71098::INFO::2014-09-25 17:46:59,402::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71098::DEBUG::2014-09-25 17:46:59,403::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`27d1f2b5-2b4b-47e6-a978-c475852b70fd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71098::DEBUG::2014-09-25 17:46:59,403::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71098::DEBUG::2014-09-25 17:46:59,403::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71098::DEBUG::2014-09-25 17:46:59,404::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`27d1f2b5-2b4b-47e6-a978-c475852b70fd`::Granted request >Thread-71098::DEBUG::2014-09-25 17:46:59,405::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1bdd6480-37c2-4b57-bc20-bdffd5cc71f4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71098::DEBUG::2014-09-25 17:46:59,405::task::993::Storage.TaskManager.Task::(_decref) Task=`1bdd6480-37c2-4b57-bc20-bdffd5cc71f4`::ref 1 aborting False >Thread-71098::INFO::2014-09-25 17:46:59,410::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71098::DEBUG::2014-09-25 17:46:59,410::task::1191::Storage.TaskManager.Task::(prepare) Task=`1bdd6480-37c2-4b57-bc20-bdffd5cc71f4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71098::DEBUG::2014-09-25 17:46:59,410::task::595::Storage.TaskManager.Task::(_updateState) Task=`1bdd6480-37c2-4b57-bc20-bdffd5cc71f4`::moving from state preparing -> state finished >Thread-71098::DEBUG::2014-09-25 17:46:59,411::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71098::DEBUG::2014-09-25 17:46:59,411::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71098::DEBUG::2014-09-25 17:46:59,411::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71098::DEBUG::2014-09-25 17:46:59,412::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71098::DEBUG::2014-09-25 17:46:59,412::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71098::DEBUG::2014-09-25 17:46:59,412::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71098::DEBUG::2014-09-25 17:46:59,412::task::993::Storage.TaskManager.Task::(_decref) Task=`1bdd6480-37c2-4b57-bc20-bdffd5cc71f4`::ref 0 aborting False >Thread-71098::DEBUG::2014-09-25 17:46:59,413::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71098::DEBUG::2014-09-25 17:46:59,413::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:46:59,660::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:59,660::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:59,660::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:59,660::__init__::232::IOProcess::(_processLogs) (109381) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:59,666::__init__::232::IOProcess::(_processLogs) (109381) Queuing response >Thread-79::DEBUG::2014-09-25 17:46:59,667::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:46:59,690::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00564408 s, 99.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:46:59,691::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:59,691::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:59,692::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:59,692::__init__::232::IOProcess::(_processLogs) (109382) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:46:59,697::__init__::232::IOProcess::(_processLogs) (109382) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:59,698::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:59,699::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:59,699::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:59,700::__init__::232::IOProcess::(_processLogs) (109383) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:59,700::__init__::232::IOProcess::(_processLogs) (109383) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:59,701::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:59,701::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:59,701::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:59,702::__init__::232::IOProcess::(_processLogs) (109384) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:59,702::__init__::232::IOProcess::(_processLogs) (109384) Queuing response >Thread-29::DEBUG::2014-09-25 17:46:59,703::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:46:59,703::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:46:59,704::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:46:59,704::__init__::232::IOProcess::(_processLogs) (109385) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:46:59,705::__init__::232::IOProcess::(_processLogs) (109385) Queuing response >Dummy-148::DEBUG::2014-09-25 17:46:59,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:46:59,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155426 s, 65.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:00,695::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:00,696::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71099::DEBUG::2014-09-25 17:47:00,698::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:01,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:02,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013287 s, 77.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:03,714::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:03,716::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71100::DEBUG::2014-09-25 17:47:03,718::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:04,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:47:04,028::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:04,029::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:04,029::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:04,029::__init__::232::IOProcess::(_processLogs) (109386) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:04,039::__init__::232::IOProcess::(_processLogs) (109386) Queuing response >Thread-81::DEBUG::2014-09-25 17:47:04,041::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:04,052::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0237681 s, 43.1 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 17:47:04,054::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00100046 s, 371 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:04,056::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:04,057::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:04,058::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:04,058::__init__::232::IOProcess::(_processLogs) (109387) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:04,060::__init__::232::IOProcess::(_processLogs) (109387) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:06,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:06,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132573 s, 77.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:06,261::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:06,261::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:06,262::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:06,262::__init__::232::IOProcess::(_processLogs) (109388) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:06,262::__init__::232::IOProcess::(_processLogs) (109388) Queuing response >Thread-80::DEBUG::2014-09-25 17:47:06,264::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:47:06,273::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000448178 s, 843 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:06,274::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:06,274::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:06,274::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:06,275::__init__::232::IOProcess::(_processLogs) (109389) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:06,275::__init__::232::IOProcess::(_processLogs) (109389) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:06,736::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:06,737::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71101::DEBUG::2014-09-25 17:47:06,739::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:08,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:08,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133131 s, 76.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:09,444::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71102::DEBUG::2014-09-25 17:47:09,446::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:47:09,446::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71102::DEBUG::2014-09-25 17:47:09,448::task::595::Storage.TaskManager.Task::(_updateState) Task=`52f36e06-11c5-45ab-acbb-5ffc851bab26`::moving from state init -> state preparing >Thread-71102::INFO::2014-09-25 17:47:09,448::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71102::INFO::2014-09-25 17:47:09,449::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71102::DEBUG::2014-09-25 17:47:09,449::task::1191::Storage.TaskManager.Task::(prepare) Task=`52f36e06-11c5-45ab-acbb-5ffc851bab26`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71102::DEBUG::2014-09-25 17:47:09,449::task::595::Storage.TaskManager.Task::(_updateState) Task=`52f36e06-11c5-45ab-acbb-5ffc851bab26`::moving from state preparing -> state finished >Thread-71102::DEBUG::2014-09-25 17:47:09,450::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71102::DEBUG::2014-09-25 17:47:09,450::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71102::DEBUG::2014-09-25 17:47:09,450::task::993::Storage.TaskManager.Task::(_decref) Task=`52f36e06-11c5-45ab-acbb-5ffc851bab26`::ref 0 aborting False >Thread-71102::DEBUG::2014-09-25 17:47:09,451::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71102::DEBUG::2014-09-25 17:47:09,451::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:09,464::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71103::DEBUG::2014-09-25 17:47:09,466::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:47:09,466::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71103::DEBUG::2014-09-25 17:47:09,468::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d9a1a41-ba78-408d-9cea-48d5a2e324f2`::moving from state init -> state preparing >Thread-71103::INFO::2014-09-25 17:47:09,468::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71103::DEBUG::2014-09-25 17:47:09,469::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8f4ce633-08e9-4470-8d61-82549bce4251`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71103::DEBUG::2014-09-25 17:47:09,470::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71103::DEBUG::2014-09-25 17:47:09,470::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71103::DEBUG::2014-09-25 17:47:09,471::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8f4ce633-08e9-4470-8d61-82549bce4251`::Granted request >Thread-71103::DEBUG::2014-09-25 17:47:09,471::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5d9a1a41-ba78-408d-9cea-48d5a2e324f2`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71103::DEBUG::2014-09-25 17:47:09,472::task::993::Storage.TaskManager.Task::(_decref) Task=`5d9a1a41-ba78-408d-9cea-48d5a2e324f2`::ref 1 aborting False >Thread-71103::INFO::2014-09-25 17:47:09,478::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71103::DEBUG::2014-09-25 17:47:09,478::task::1191::Storage.TaskManager.Task::(prepare) Task=`5d9a1a41-ba78-408d-9cea-48d5a2e324f2`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71103::DEBUG::2014-09-25 17:47:09,479::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d9a1a41-ba78-408d-9cea-48d5a2e324f2`::moving from state preparing -> state finished >Thread-71103::DEBUG::2014-09-25 17:47:09,479::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71103::DEBUG::2014-09-25 17:47:09,479::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71103::DEBUG::2014-09-25 17:47:09,480::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71103::DEBUG::2014-09-25 17:47:09,480::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71103::DEBUG::2014-09-25 17:47:09,480::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71103::DEBUG::2014-09-25 17:47:09,481::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71103::DEBUG::2014-09-25 17:47:09,481::task::993::Storage.TaskManager.Task::(_decref) Task=`5d9a1a41-ba78-408d-9cea-48d5a2e324f2`::ref 0 aborting False >Thread-71103::DEBUG::2014-09-25 17:47:09,482::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71103::DEBUG::2014-09-25 17:47:09,482::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:09,705::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:09,705::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:09,706::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:09,707::__init__::232::IOProcess::(_processLogs) (109390) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 17:47:09,707::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:47:09,708::__init__::232::IOProcess::(_processLogs) (109390) Queuing response >Thread-79::DEBUG::2014-09-25 17:47:09,717::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000399778 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:09,718::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:09,719::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:09,719::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:09,719::__init__::232::IOProcess::(_processLogs) (109391) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:09,720::__init__::232::IOProcess::(_processLogs) (109391) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:09,721::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:09,721::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:09,721::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:09,722::__init__::232::IOProcess::(_processLogs) (109392) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:09,722::__init__::232::IOProcess::(_processLogs) (109392) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:09,723::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:09,723::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:09,724::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:09,724::__init__::232::IOProcess::(_processLogs) (109393) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:09,725::__init__::232::IOProcess::(_processLogs) (109393) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:09,725::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:09,725::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:09,726::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:09,727::__init__::232::IOProcess::(_processLogs) (109394) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:09,727::__init__::232::IOProcess::(_processLogs) (109394) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:09,755::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:09,756::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71104::DEBUG::2014-09-25 17:47:09,759::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b0d868a-8be7-4e80-b670-ae3dd5f41180`::moving from state init -> state preparing >Thread-71104::INFO::2014-09-25 17:47:09,759::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71104::INFO::2014-09-25 17:47:09,760::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000399778', 'lastCheck': '0.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000448178', 'lastCheck': '3.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00100046', 'lastCheck': '5.7', 'valid': True}} >Thread-71104::DEBUG::2014-09-25 17:47:09,760::task::1191::Storage.TaskManager.Task::(prepare) Task=`0b0d868a-8be7-4e80-b670-ae3dd5f41180`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000399778', 'lastCheck': '0.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000448178', 'lastCheck': '3.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00100046', 'lastCheck': '5.7', 'valid': True}} >Thread-71104::DEBUG::2014-09-25 17:47:09,760::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b0d868a-8be7-4e80-b670-ae3dd5f41180`::moving from state preparing -> state finished >Thread-71104::DEBUG::2014-09-25 17:47:09,761::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71104::DEBUG::2014-09-25 17:47:09,761::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71104::DEBUG::2014-09-25 17:47:09,761::task::993::Storage.TaskManager.Task::(_decref) Task=`0b0d868a-8be7-4e80-b670-ae3dd5f41180`::ref 0 aborting False >Thread-71104::DEBUG::2014-09-25 17:47:09,763::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:09,783::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:09,784::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71105::DEBUG::2014-09-25 17:47:09,786::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:10,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:10,188::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132734 s, 77.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:47:12,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:12,238::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160717 s, 63.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:12,848::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:12,849::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71106::DEBUG::2014-09-25 17:47:12,851::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:14,063::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:14,063::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:14,063::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:14,064::__init__::232::IOProcess::(_processLogs) (109395) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:14,064::__init__::232::IOProcess::(_processLogs) (109395) Queuing response >Thread-81::DEBUG::2014-09-25 17:47:14,065::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:47:14,090::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00737016 s, 50.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:14,091::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:14,092::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:14,092::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:14,092::__init__::232::IOProcess::(_processLogs) (109396) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:14,097::__init__::232::IOProcess::(_processLogs) (109396) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:14,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:14,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159725 s, 64.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:15,868::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:15,870::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71107::DEBUG::2014-09-25 17:47:15,872::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:16,278::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:16,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:16,281::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:16,282::__init__::232::IOProcess::(_processLogs) (109397) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:16,284::__init__::232::IOProcess::(_processLogs) (109397) Queuing response >Thread-80::DEBUG::2014-09-25 17:47:16,286::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:47:16,297::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000584226 s, 647 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:16,300::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:16,301::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:16,303::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:16,304::__init__::232::IOProcess::(_processLogs) (109398) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:16,306::__init__::232::IOProcess::(_processLogs) (109398) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:16,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:16,343::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0226371 s, 45.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:47:18,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:18,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136506 s, 75.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:18,888::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:18,890::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71108::DEBUG::2014-09-25 17:47:18,892::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:19,512::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71109::DEBUG::2014-09-25 17:47:19,514::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:47:19,514::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71109::DEBUG::2014-09-25 17:47:19,516::task::595::Storage.TaskManager.Task::(_updateState) Task=`2a57a66a-2bc6-40fa-a74d-1a701c5bd24b`::moving from state init -> state preparing >Thread-71109::INFO::2014-09-25 17:47:19,516::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71109::INFO::2014-09-25 17:47:19,517::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71109::DEBUG::2014-09-25 17:47:19,517::task::1191::Storage.TaskManager.Task::(prepare) Task=`2a57a66a-2bc6-40fa-a74d-1a701c5bd24b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71109::DEBUG::2014-09-25 17:47:19,517::task::595::Storage.TaskManager.Task::(_updateState) Task=`2a57a66a-2bc6-40fa-a74d-1a701c5bd24b`::moving from state preparing -> state finished >Thread-71109::DEBUG::2014-09-25 17:47:19,517::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71109::DEBUG::2014-09-25 17:47:19,518::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71109::DEBUG::2014-09-25 17:47:19,518::task::993::Storage.TaskManager.Task::(_decref) Task=`2a57a66a-2bc6-40fa-a74d-1a701c5bd24b`::ref 0 aborting False >Thread-71109::DEBUG::2014-09-25 17:47:19,518::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71109::DEBUG::2014-09-25 17:47:19,519::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:19,533::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:19,534::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71110::DEBUG::2014-09-25 17:47:19,534::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71110::DEBUG::2014-09-25 17:47:19,535::task::595::Storage.TaskManager.Task::(_updateState) Task=`b85875a6-44bd-4519-8243-9becdd842845`::moving from state init -> state preparing >Thread-71110::INFO::2014-09-25 17:47:19,536::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71110::DEBUG::2014-09-25 17:47:19,536::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`dd9ac1df-4e17-49e3-add4-8fbaf05c7866`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71110::DEBUG::2014-09-25 17:47:19,537::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71110::DEBUG::2014-09-25 17:47:19,537::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71110::DEBUG::2014-09-25 17:47:19,537::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`dd9ac1df-4e17-49e3-add4-8fbaf05c7866`::Granted request >Thread-71110::DEBUG::2014-09-25 17:47:19,538::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b85875a6-44bd-4519-8243-9becdd842845`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71110::DEBUG::2014-09-25 17:47:19,538::task::993::Storage.TaskManager.Task::(_decref) Task=`b85875a6-44bd-4519-8243-9becdd842845`::ref 1 aborting False >Thread-71110::INFO::2014-09-25 17:47:19,542::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71110::DEBUG::2014-09-25 17:47:19,543::task::1191::Storage.TaskManager.Task::(prepare) Task=`b85875a6-44bd-4519-8243-9becdd842845`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71110::DEBUG::2014-09-25 17:47:19,543::task::595::Storage.TaskManager.Task::(_updateState) Task=`b85875a6-44bd-4519-8243-9becdd842845`::moving from state preparing -> state finished >Thread-71110::DEBUG::2014-09-25 17:47:19,543::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71110::DEBUG::2014-09-25 17:47:19,543::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71110::DEBUG::2014-09-25 17:47:19,544::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71110::DEBUG::2014-09-25 17:47:19,544::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71110::DEBUG::2014-09-25 17:47:19,544::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71110::DEBUG::2014-09-25 17:47:19,545::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71110::DEBUG::2014-09-25 17:47:19,545::task::993::Storage.TaskManager.Task::(_decref) Task=`b85875a6-44bd-4519-8243-9becdd842845`::ref 0 aborting False >Thread-71110::DEBUG::2014-09-25 17:47:19,545::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71110::DEBUG::2014-09-25 17:47:19,546::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:19,727::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:19,728::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:19,728::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:19,728::__init__::232::IOProcess::(_processLogs) (109399) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:19,729::__init__::232::IOProcess::(_processLogs) (109399) Queuing response >Thread-79::DEBUG::2014-09-25 17:47:19,730::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:47:19,738::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000361863 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:19,739::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:19,740::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:19,740::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:19,740::__init__::232::IOProcess::(_processLogs) (109400) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:19,741::__init__::232::IOProcess::(_processLogs) (109400) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:19,742::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:19,742::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:19,743::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:19,744::__init__::232::IOProcess::(_processLogs) (109401) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:19,744::__init__::232::IOProcess::(_processLogs) (109401) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:19,744::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:19,745::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:19,745::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:19,746::__init__::232::IOProcess::(_processLogs) (109402) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:19,746::__init__::232::IOProcess::(_processLogs) (109402) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:19,746::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:19,747::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:19,747::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:19,748::__init__::232::IOProcess::(_processLogs) (109403) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:19,749::__init__::232::IOProcess::(_processLogs) (109403) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:20,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:20,617::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133027 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:21,907::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:21,909::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71111::DEBUG::2014-09-25 17:47:21,911::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:22,638::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:22,676::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0275602 s, 37.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:24,099::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:24,099::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:24,099::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:24,100::__init__::232::IOProcess::(_processLogs) (109404) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:24,100::__init__::232::IOProcess::(_processLogs) (109404) Queuing response >Thread-81::DEBUG::2014-09-25 17:47:24,101::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:47:24,110::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000403425 s, 920 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:24,112::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:24,112::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:24,112::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:24,112::__init__::232::IOProcess::(_processLogs) (109405) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:24,113::__init__::232::IOProcess::(_processLogs) (109405) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:24,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:24,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013468 s, 76.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:24,925::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:24,927::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71112::DEBUG::2014-09-25 17:47:24,929::task::595::Storage.TaskManager.Task::(_updateState) Task=`1f6c1a3f-5a75-49c7-b813-d8f8d2d3b7d6`::moving from state init -> state preparing >Thread-71112::INFO::2014-09-25 17:47:24,929::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71112::INFO::2014-09-25 17:47:24,930::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000361863', 'lastCheck': '5.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000584226', 'lastCheck': '8.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000403425', 'lastCheck': '0.8', 'valid': True}} >Thread-71112::DEBUG::2014-09-25 17:47:24,930::task::1191::Storage.TaskManager.Task::(prepare) Task=`1f6c1a3f-5a75-49c7-b813-d8f8d2d3b7d6`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000361863', 'lastCheck': '5.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000584226', 'lastCheck': '8.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000403425', 'lastCheck': '0.8', 'valid': True}} >Thread-71112::DEBUG::2014-09-25 17:47:24,930::task::595::Storage.TaskManager.Task::(_updateState) Task=`1f6c1a3f-5a75-49c7-b813-d8f8d2d3b7d6`::moving from state preparing -> state finished >Thread-71112::DEBUG::2014-09-25 17:47:24,931::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71112::DEBUG::2014-09-25 17:47:24,931::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71112::DEBUG::2014-09-25 17:47:24,931::task::993::Storage.TaskManager.Task::(_decref) Task=`1f6c1a3f-5a75-49c7-b813-d8f8d2d3b7d6`::ref 0 aborting False >Thread-71112::DEBUG::2014-09-25 17:47:24,933::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:24,952::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:24,953::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71113::DEBUG::2014-09-25 17:47:24,955::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:26,310::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:26,310::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:26,310::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:26,311::__init__::232::IOProcess::(_processLogs) (109406) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:26,311::__init__::232::IOProcess::(_processLogs) (109406) Queuing response >Thread-80::DEBUG::2014-09-25 17:47:26,312::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:47:26,321::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000448477 s, 843 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:26,322::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:26,323::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:26,323::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:26,323::__init__::232::IOProcess::(_processLogs) (109407) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:26,323::__init__::232::IOProcess::(_processLogs) (109407) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:26,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:26,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151398 s, 67.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:28,018::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:28,019::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71114::DEBUG::2014-09-25 17:47:28,021::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:28,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:28,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132203 s, 77.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:29,578::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:29,579::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71115::DEBUG::2014-09-25 17:47:29,579::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71115::DEBUG::2014-09-25 17:47:29,581::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a9b73ab-8efe-4b03-8cfe-5c27fff4be74`::moving from state init -> state preparing >Thread-71115::INFO::2014-09-25 17:47:29,582::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71115::INFO::2014-09-25 17:47:29,582::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71115::DEBUG::2014-09-25 17:47:29,583::task::1191::Storage.TaskManager.Task::(prepare) Task=`1a9b73ab-8efe-4b03-8cfe-5c27fff4be74`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71115::DEBUG::2014-09-25 17:47:29,583::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a9b73ab-8efe-4b03-8cfe-5c27fff4be74`::moving from state preparing -> state finished >Thread-71115::DEBUG::2014-09-25 17:47:29,583::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71115::DEBUG::2014-09-25 17:47:29,583::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71115::DEBUG::2014-09-25 17:47:29,583::task::993::Storage.TaskManager.Task::(_decref) Task=`1a9b73ab-8efe-4b03-8cfe-5c27fff4be74`::ref 0 aborting False >Thread-71115::DEBUG::2014-09-25 17:47:29,584::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71115::DEBUG::2014-09-25 17:47:29,584::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:29,598::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71116::DEBUG::2014-09-25 17:47:29,600::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:47:29,599::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71116::DEBUG::2014-09-25 17:47:29,602::task::595::Storage.TaskManager.Task::(_updateState) Task=`e1072945-9531-4b20-a1ed-812826e55ca1`::moving from state init -> state preparing >Thread-71116::INFO::2014-09-25 17:47:29,602::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71116::DEBUG::2014-09-25 17:47:29,602::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ff803099-e379-4449-9c0a-9d47aa4d9d32`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71116::DEBUG::2014-09-25 17:47:29,603::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71116::DEBUG::2014-09-25 17:47:29,603::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71116::DEBUG::2014-09-25 17:47:29,603::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ff803099-e379-4449-9c0a-9d47aa4d9d32`::Granted request >Thread-71116::DEBUG::2014-09-25 17:47:29,604::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e1072945-9531-4b20-a1ed-812826e55ca1`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71116::DEBUG::2014-09-25 17:47:29,604::task::993::Storage.TaskManager.Task::(_decref) Task=`e1072945-9531-4b20-a1ed-812826e55ca1`::ref 1 aborting False >Thread-71116::INFO::2014-09-25 17:47:29,609::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71116::DEBUG::2014-09-25 17:47:29,609::task::1191::Storage.TaskManager.Task::(prepare) Task=`e1072945-9531-4b20-a1ed-812826e55ca1`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71116::DEBUG::2014-09-25 17:47:29,609::task::595::Storage.TaskManager.Task::(_updateState) Task=`e1072945-9531-4b20-a1ed-812826e55ca1`::moving from state preparing -> state finished >Thread-71116::DEBUG::2014-09-25 17:47:29,609::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71116::DEBUG::2014-09-25 17:47:29,610::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71116::DEBUG::2014-09-25 17:47:29,610::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71116::DEBUG::2014-09-25 17:47:29,610::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71116::DEBUG::2014-09-25 17:47:29,610::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71116::DEBUG::2014-09-25 17:47:29,611::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71116::DEBUG::2014-09-25 17:47:29,611::task::993::Storage.TaskManager.Task::(_decref) Task=`e1072945-9531-4b20-a1ed-812826e55ca1`::ref 0 aborting False >Thread-71116::DEBUG::2014-09-25 17:47:29,611::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71116::DEBUG::2014-09-25 17:47:29,612::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:29,749::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:29,749::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:29,750::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:29,750::__init__::232::IOProcess::(_processLogs) (109408) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:29,750::__init__::232::IOProcess::(_processLogs) (109408) Queuing response >Thread-79::DEBUG::2014-09-25 17:47:29,751::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:47:29,760::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00044497 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:29,762::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:29,762::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:29,763::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:29,764::__init__::232::IOProcess::(_processLogs) (109409) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:29,764::__init__::232::IOProcess::(_processLogs) (109409) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:29,765::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:29,765::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:29,765::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:29,766::__init__::232::IOProcess::(_processLogs) (109410) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:29,766::__init__::232::IOProcess::(_processLogs) (109410) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:29,766::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:29,767::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:29,767::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:29,768::__init__::232::IOProcess::(_processLogs) (109411) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:29,768::__init__::232::IOProcess::(_processLogs) (109411) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:29,769::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:29,769::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:29,769::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:29,770::__init__::232::IOProcess::(_processLogs) (109412) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:29,770::__init__::232::IOProcess::(_processLogs) (109412) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:30,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:30,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135651 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:31,041::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:31,043::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71117::DEBUG::2014-09-25 17:47:31,045::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:32,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:32,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152595 s, 67.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:34,062::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:34,063::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71118::DEBUG::2014-09-25 17:47:34,066::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 17:47:34,114::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:47:34,118::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:34,119::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:34,119::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:34,119::__init__::232::IOProcess::(_processLogs) (109413) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:47:34,120::__init__::232::IOProcess::(_processLogs) (109413) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:34,122::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:34,124::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:34,125::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:34,127::__init__::232::IOProcess::(_processLogs) (109414) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:47:34,130::__init__::232::IOProcess::(_processLogs) (109414) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:34,132::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:34,133::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-81::DEBUG::2014-09-25 17:47:34,134::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:47:34,135::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:34,136::__init__::232::IOProcess::(_processLogs) (109415) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:47:34,136::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:34,136::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:34,137::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:34,137::__init__::232::IOProcess::(_processLogs) (109416) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:47:34,137::__init__::232::IOProcess::(_processLogs) (109415) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:34,213::__init__::232::IOProcess::(_processLogs) (109416) Queuing response >Thread-81::DEBUG::2014-09-25 17:47:34,214::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:47:34,215::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:34,215::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:34,216::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:34,216::__init__::232::IOProcess::(_processLogs) (109417) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:47:34,216::__init__::232::IOProcess::(_processLogs) (109417) Queuing response >Thread-81::DEBUG::2014-09-25 17:47:34,216::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:47:34,218::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:34,218::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:34,219::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:34,219::__init__::232::IOProcess::(_processLogs) (109418) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:34,220::__init__::232::IOProcess::(_processLogs) (109418) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:34,220::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:34,220::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:34,221::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:34,221::__init__::232::IOProcess::(_processLogs) (109419) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:47:34,222::__init__::232::IOProcess::(_processLogs) (109419) Queuing response >Thread-81::DEBUG::2014-09-25 17:47:34,222::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 17:47:34,222::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:47:34,223::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:47:34,223::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:34,224::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:34,224::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:34,224::__init__::232::IOProcess::(_processLogs) (109420) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:34,224::__init__::232::IOProcess::(_processLogs) (109420) Queuing response >Thread-81::DEBUG::2014-09-25 17:47:34,226::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:47:34,239::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000472239 s, 786 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:34,240::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:34,240::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:34,241::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:34,241::__init__::232::IOProcess::(_processLogs) (109421) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:34,247::__init__::232::IOProcess::(_processLogs) (109421) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:35,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:35,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013515 s, 75.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:36,326::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:36,326::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:36,326::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:36,327::__init__::232::IOProcess::(_processLogs) (109422) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:36,327::__init__::232::IOProcess::(_processLogs) (109422) Queuing response >Thread-80::DEBUG::2014-09-25 17:47:36,329::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:47:36,337::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000540001 s, 700 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:36,339::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:36,339::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:36,339::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:36,339::__init__::232::IOProcess::(_processLogs) (109423) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:36,340::__init__::232::IOProcess::(_processLogs) (109423) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:37,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:37,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133141 s, 76.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:37,084::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:37,087::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71122::DEBUG::2014-09-25 17:47:37,092::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:39,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:39,122::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135147 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:39,640::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:39,642::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71123::DEBUG::2014-09-25 17:47:39,642::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71123::DEBUG::2014-09-25 17:47:39,644::task::595::Storage.TaskManager.Task::(_updateState) Task=`fde0ccbb-a16d-41ff-8e6f-0ffc50e83b55`::moving from state init -> state preparing >Thread-71123::INFO::2014-09-25 17:47:39,645::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71123::INFO::2014-09-25 17:47:39,645::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71123::DEBUG::2014-09-25 17:47:39,646::task::1191::Storage.TaskManager.Task::(prepare) Task=`fde0ccbb-a16d-41ff-8e6f-0ffc50e83b55`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71123::DEBUG::2014-09-25 17:47:39,646::task::595::Storage.TaskManager.Task::(_updateState) Task=`fde0ccbb-a16d-41ff-8e6f-0ffc50e83b55`::moving from state preparing -> state finished >Thread-71123::DEBUG::2014-09-25 17:47:39,646::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71123::DEBUG::2014-09-25 17:47:39,646::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71123::DEBUG::2014-09-25 17:47:39,646::task::993::Storage.TaskManager.Task::(_decref) Task=`fde0ccbb-a16d-41ff-8e6f-0ffc50e83b55`::ref 0 aborting False >Thread-71123::DEBUG::2014-09-25 17:47:39,647::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71123::DEBUG::2014-09-25 17:47:39,647::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:39,661::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71124::DEBUG::2014-09-25 17:47:39,662::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:47:39,662::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71124::DEBUG::2014-09-25 17:47:39,664::task::595::Storage.TaskManager.Task::(_updateState) Task=`12d1791a-53ba-4309-9135-585b756af2cd`::moving from state init -> state preparing >Thread-71124::INFO::2014-09-25 17:47:39,664::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71124::DEBUG::2014-09-25 17:47:39,665::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c889fd1e-7c07-452b-961a-cda771c2219f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71124::DEBUG::2014-09-25 17:47:39,665::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71124::DEBUG::2014-09-25 17:47:39,666::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71124::DEBUG::2014-09-25 17:47:39,666::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c889fd1e-7c07-452b-961a-cda771c2219f`::Granted request >Thread-71124::DEBUG::2014-09-25 17:47:39,666::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`12d1791a-53ba-4309-9135-585b756af2cd`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71124::DEBUG::2014-09-25 17:47:39,667::task::993::Storage.TaskManager.Task::(_decref) Task=`12d1791a-53ba-4309-9135-585b756af2cd`::ref 1 aborting False >Thread-71124::INFO::2014-09-25 17:47:39,671::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71124::DEBUG::2014-09-25 17:47:39,671::task::1191::Storage.TaskManager.Task::(prepare) Task=`12d1791a-53ba-4309-9135-585b756af2cd`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71124::DEBUG::2014-09-25 17:47:39,672::task::595::Storage.TaskManager.Task::(_updateState) Task=`12d1791a-53ba-4309-9135-585b756af2cd`::moving from state preparing -> state finished >Thread-71124::DEBUG::2014-09-25 17:47:39,672::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71124::DEBUG::2014-09-25 17:47:39,672::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71124::DEBUG::2014-09-25 17:47:39,673::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71124::DEBUG::2014-09-25 17:47:39,673::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71124::DEBUG::2014-09-25 17:47:39,673::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71124::DEBUG::2014-09-25 17:47:39,673::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71124::DEBUG::2014-09-25 17:47:39,674::task::993::Storage.TaskManager.Task::(_decref) Task=`12d1791a-53ba-4309-9135-585b756af2cd`::ref 0 aborting False >Thread-71124::DEBUG::2014-09-25 17:47:39,674::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71124::DEBUG::2014-09-25 17:47:39,675::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:39,771::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:39,771::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:39,771::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:39,772::__init__::232::IOProcess::(_processLogs) (109424) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:39,772::__init__::232::IOProcess::(_processLogs) (109424) Queuing response >Thread-79::DEBUG::2014-09-25 17:47:39,773::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:47:39,782::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000518756 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:39,783::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:39,783::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:39,784::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:39,784::__init__::232::IOProcess::(_processLogs) (109425) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:39,785::__init__::232::IOProcess::(_processLogs) (109425) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:39,786::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:39,786::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:39,787::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:39,787::__init__::232::IOProcess::(_processLogs) (109426) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:39,787::__init__::232::IOProcess::(_processLogs) (109426) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:39,788::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:39,788::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:39,789::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:39,789::__init__::232::IOProcess::(_processLogs) (109427) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:39,790::__init__::232::IOProcess::(_processLogs) (109427) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:39,790::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:39,790::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:39,791::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:39,792::__init__::232::IOProcess::(_processLogs) (109428) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:39,792::__init__::232::IOProcess::(_processLogs) (109428) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:40,109::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:40,111::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71125::DEBUG::2014-09-25 17:47:40,114::task::595::Storage.TaskManager.Task::(_updateState) Task=`040ab14f-88b2-4294-a778-e12867ee491c`::moving from state init -> state preparing >Thread-71125::INFO::2014-09-25 17:47:40,114::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71125::INFO::2014-09-25 17:47:40,115::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000518756', 'lastCheck': '0.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000540001', 'lastCheck': '3.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000472239', 'lastCheck': '5.9', 'valid': True}} >Thread-71125::DEBUG::2014-09-25 17:47:40,115::task::1191::Storage.TaskManager.Task::(prepare) Task=`040ab14f-88b2-4294-a778-e12867ee491c`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000518756', 'lastCheck': '0.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000540001', 'lastCheck': '3.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000472239', 'lastCheck': '5.9', 'valid': True}} >Thread-71125::DEBUG::2014-09-25 17:47:40,115::task::595::Storage.TaskManager.Task::(_updateState) Task=`040ab14f-88b2-4294-a778-e12867ee491c`::moving from state preparing -> state finished >Thread-71125::DEBUG::2014-09-25 17:47:40,115::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71125::DEBUG::2014-09-25 17:47:40,116::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71125::DEBUG::2014-09-25 17:47:40,116::task::993::Storage.TaskManager.Task::(_decref) Task=`040ab14f-88b2-4294-a778-e12867ee491c`::ref 0 aborting False >Thread-71125::DEBUG::2014-09-25 17:47:40,118::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:40,142::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:40,144::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71126::DEBUG::2014-09-25 17:47:40,146::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:41,142::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:41,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135004 s, 75.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:47:43,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:43,206::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013425 s, 76.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:43,210::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:43,214::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71127::DEBUG::2014-09-25 17:47:43,218::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:44,249::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:44,249::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:44,250::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:44,250::__init__::232::IOProcess::(_processLogs) (109429) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:44,250::__init__::232::IOProcess::(_processLogs) (109429) Queuing response >Thread-81::DEBUG::2014-09-25 17:47:44,252::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:47:44,260::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000441725 s, 840 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:44,263::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:44,264::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:44,265::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:44,266::__init__::232::IOProcess::(_processLogs) (109430) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:44,266::__init__::232::IOProcess::(_processLogs) (109430) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:45,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:45,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136054 s, 75.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:46,238::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:46,241::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71128::DEBUG::2014-09-25 17:47:46,247::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:46,344::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:46,345::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:46,346::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:46,347::__init__::232::IOProcess::(_processLogs) (109431) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:46,349::__init__::232::IOProcess::(_processLogs) (109431) Queuing response >Thread-80::DEBUG::2014-09-25 17:47:46,351::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:47:46,362::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000591447 s, 639 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:46,365::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:46,367::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:46,368::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:46,369::__init__::232::IOProcess::(_processLogs) (109432) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:46,371::__init__::232::IOProcess::(_processLogs) (109432) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:47,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:47,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0122509 s, 83.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:49,267::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:49,268::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71129::DEBUG::2014-09-25 17:47:49,270::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:49,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:49,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132445 s, 77.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:49,705::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:49,707::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71130::DEBUG::2014-09-25 17:47:49,707::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71130::DEBUG::2014-09-25 17:47:49,709::task::595::Storage.TaskManager.Task::(_updateState) Task=`9bafb89a-a5e7-4011-897d-09deb13c0104`::moving from state init -> state preparing >Thread-71130::INFO::2014-09-25 17:47:49,710::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71130::INFO::2014-09-25 17:47:49,710::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71130::DEBUG::2014-09-25 17:47:49,710::task::1191::Storage.TaskManager.Task::(prepare) Task=`9bafb89a-a5e7-4011-897d-09deb13c0104`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71130::DEBUG::2014-09-25 17:47:49,711::task::595::Storage.TaskManager.Task::(_updateState) Task=`9bafb89a-a5e7-4011-897d-09deb13c0104`::moving from state preparing -> state finished >Thread-71130::DEBUG::2014-09-25 17:47:49,711::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71130::DEBUG::2014-09-25 17:47:49,711::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71130::DEBUG::2014-09-25 17:47:49,711::task::993::Storage.TaskManager.Task::(_decref) Task=`9bafb89a-a5e7-4011-897d-09deb13c0104`::ref 0 aborting False >Thread-71130::DEBUG::2014-09-25 17:47:49,712::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71130::DEBUG::2014-09-25 17:47:49,712::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:49,729::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71131::DEBUG::2014-09-25 17:47:49,730::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:47:49,730::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71131::DEBUG::2014-09-25 17:47:49,732::task::595::Storage.TaskManager.Task::(_updateState) Task=`5711eca5-2221-4b8a-937e-4fc80499580b`::moving from state init -> state preparing >Thread-71131::INFO::2014-09-25 17:47:49,733::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71131::DEBUG::2014-09-25 17:47:49,733::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2eba792e-f220-4f0f-98b1-c28d93cd0b3b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71131::DEBUG::2014-09-25 17:47:49,733::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71131::DEBUG::2014-09-25 17:47:49,734::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71131::DEBUG::2014-09-25 17:47:49,734::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2eba792e-f220-4f0f-98b1-c28d93cd0b3b`::Granted request >Thread-71131::DEBUG::2014-09-25 17:47:49,735::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5711eca5-2221-4b8a-937e-4fc80499580b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71131::DEBUG::2014-09-25 17:47:49,735::task::993::Storage.TaskManager.Task::(_decref) Task=`5711eca5-2221-4b8a-937e-4fc80499580b`::ref 1 aborting False >Thread-71131::INFO::2014-09-25 17:47:49,739::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71131::DEBUG::2014-09-25 17:47:49,740::task::1191::Storage.TaskManager.Task::(prepare) Task=`5711eca5-2221-4b8a-937e-4fc80499580b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71131::DEBUG::2014-09-25 17:47:49,740::task::595::Storage.TaskManager.Task::(_updateState) Task=`5711eca5-2221-4b8a-937e-4fc80499580b`::moving from state preparing -> state finished >Thread-71131::DEBUG::2014-09-25 17:47:49,740::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71131::DEBUG::2014-09-25 17:47:49,740::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71131::DEBUG::2014-09-25 17:47:49,741::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71131::DEBUG::2014-09-25 17:47:49,741::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71131::DEBUG::2014-09-25 17:47:49,741::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71131::DEBUG::2014-09-25 17:47:49,741::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71131::DEBUG::2014-09-25 17:47:49,742::task::993::Storage.TaskManager.Task::(_decref) Task=`5711eca5-2221-4b8a-937e-4fc80499580b`::ref 0 aborting False >Thread-71131::DEBUG::2014-09-25 17:47:49,742::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71131::DEBUG::2014-09-25 17:47:49,743::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:49,792::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:49,792::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:49,793::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:49,793::__init__::232::IOProcess::(_processLogs) (109433) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:49,794::__init__::232::IOProcess::(_processLogs) (109433) Queuing response >Thread-79::DEBUG::2014-09-25 17:47:49,795::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:47:49,810::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00408219 s, 138 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:49,811::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:49,811::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:49,812::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:49,812::__init__::232::IOProcess::(_processLogs) (109434) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:49,813::__init__::232::IOProcess::(_processLogs) (109434) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:49,814::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:49,815::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:49,815::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:49,815::__init__::232::IOProcess::(_processLogs) (109435) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:49,819::__init__::232::IOProcess::(_processLogs) (109435) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:49,820::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:49,820::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:49,821::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:49,821::__init__::232::IOProcess::(_processLogs) (109436) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:49,821::__init__::232::IOProcess::(_processLogs) (109436) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:49,822::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:49,822::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:49,823::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:49,823::__init__::232::IOProcess::(_processLogs) (109437) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:49,825::__init__::232::IOProcess::(_processLogs) (109437) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:51,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:51,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155171 s, 66.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:52,286::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:52,288::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71132::DEBUG::2014-09-25 17:47:52,290::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:53,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:53,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0176816 s, 57.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:54,267::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:54,267::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:54,267::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:54,268::__init__::232::IOProcess::(_processLogs) (109438) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:54,268::__init__::232::IOProcess::(_processLogs) (109438) Queuing response >Thread-81::DEBUG::2014-09-25 17:47:54,269::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:47:54,278::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000351218 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:54,279::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:54,279::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:54,280::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:54,282::__init__::232::IOProcess::(_processLogs) (109439) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:54,282::__init__::232::IOProcess::(_processLogs) (109439) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:55,305::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:55,306::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71133::DEBUG::2014-09-25 17:47:55,309::task::595::Storage.TaskManager.Task::(_updateState) Task=`1e94d69e-ad51-4495-9818-0e6d844e3751`::moving from state init -> state preparing >Thread-71133::INFO::2014-09-25 17:47:55,309::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71133::INFO::2014-09-25 17:47:55,310::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00408219', 'lastCheck': '5.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000591447', 'lastCheck': '8.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000351218', 'lastCheck': '1.0', 'valid': True}} >Thread-71133::DEBUG::2014-09-25 17:47:55,310::task::1191::Storage.TaskManager.Task::(prepare) Task=`1e94d69e-ad51-4495-9818-0e6d844e3751`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00408219', 'lastCheck': '5.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000591447', 'lastCheck': '8.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000351218', 'lastCheck': '1.0', 'valid': True}} >Thread-71133::DEBUG::2014-09-25 17:47:55,310::task::595::Storage.TaskManager.Task::(_updateState) Task=`1e94d69e-ad51-4495-9818-0e6d844e3751`::moving from state preparing -> state finished >Thread-71133::DEBUG::2014-09-25 17:47:55,310::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71133::DEBUG::2014-09-25 17:47:55,311::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71133::DEBUG::2014-09-25 17:47:55,311::task::993::Storage.TaskManager.Task::(_decref) Task=`1e94d69e-ad51-4495-9818-0e6d844e3751`::ref 0 aborting False >Thread-71133::DEBUG::2014-09-25 17:47:55,313::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:55,335::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:55,336::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71134::DEBUG::2014-09-25 17:47:55,338::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:55,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:55,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136921 s, 74.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:56,374::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:56,375::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:56,375::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:56,375::__init__::232::IOProcess::(_processLogs) (109440) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:56,376::__init__::232::IOProcess::(_processLogs) (109440) Queuing response >Thread-80::DEBUG::2014-09-25 17:47:56,377::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:47:56,386::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000436981 s, 865 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:56,387::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:56,388::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:56,388::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:56,388::__init__::232::IOProcess::(_processLogs) (109441) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:56,389::__init__::232::IOProcess::(_processLogs) (109441) Queuing response >Dummy-148::DEBUG::2014-09-25 17:47:57,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:57,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133842 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:58,405::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:58,406::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71135::DEBUG::2014-09-25 17:47:58,408::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:47:59,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:47:59,553::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137826 s, 74.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:59,770::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71136::DEBUG::2014-09-25 17:47:59,772::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:47:59,772::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71136::DEBUG::2014-09-25 17:47:59,774::task::595::Storage.TaskManager.Task::(_updateState) Task=`193657e9-11b7-4607-a5d3-b769d917ebe5`::moving from state init -> state preparing >Thread-71136::INFO::2014-09-25 17:47:59,775::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71136::INFO::2014-09-25 17:47:59,775::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71136::DEBUG::2014-09-25 17:47:59,776::task::1191::Storage.TaskManager.Task::(prepare) Task=`193657e9-11b7-4607-a5d3-b769d917ebe5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71136::DEBUG::2014-09-25 17:47:59,776::task::595::Storage.TaskManager.Task::(_updateState) Task=`193657e9-11b7-4607-a5d3-b769d917ebe5`::moving from state preparing -> state finished >Thread-71136::DEBUG::2014-09-25 17:47:59,776::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71136::DEBUG::2014-09-25 17:47:59,776::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71136::DEBUG::2014-09-25 17:47:59,776::task::993::Storage.TaskManager.Task::(_decref) Task=`193657e9-11b7-4607-a5d3-b769d917ebe5`::ref 0 aborting False >Thread-71136::DEBUG::2014-09-25 17:47:59,777::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71136::DEBUG::2014-09-25 17:47:59,777::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:47:59,792::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:47:59,793::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71137::DEBUG::2014-09-25 17:47:59,793::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71137::DEBUG::2014-09-25 17:47:59,795::task::595::Storage.TaskManager.Task::(_updateState) Task=`5581a2a5-d95b-41fe-b2f0-194786b0a5c0`::moving from state init -> state preparing >Thread-71137::INFO::2014-09-25 17:47:59,795::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71137::DEBUG::2014-09-25 17:47:59,796::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`666905cd-72cb-4b78-964e-8ae76ea2d351`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71137::DEBUG::2014-09-25 17:47:59,796::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71137::DEBUG::2014-09-25 17:47:59,797::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71137::DEBUG::2014-09-25 17:47:59,797::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`666905cd-72cb-4b78-964e-8ae76ea2d351`::Granted request >Thread-71137::DEBUG::2014-09-25 17:47:59,797::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5581a2a5-d95b-41fe-b2f0-194786b0a5c0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71137::DEBUG::2014-09-25 17:47:59,798::task::993::Storage.TaskManager.Task::(_decref) Task=`5581a2a5-d95b-41fe-b2f0-194786b0a5c0`::ref 1 aborting False >Thread-71137::INFO::2014-09-25 17:47:59,802::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71137::DEBUG::2014-09-25 17:47:59,802::task::1191::Storage.TaskManager.Task::(prepare) Task=`5581a2a5-d95b-41fe-b2f0-194786b0a5c0`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71137::DEBUG::2014-09-25 17:47:59,803::task::595::Storage.TaskManager.Task::(_updateState) Task=`5581a2a5-d95b-41fe-b2f0-194786b0a5c0`::moving from state preparing -> state finished >Thread-71137::DEBUG::2014-09-25 17:47:59,803::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71137::DEBUG::2014-09-25 17:47:59,803::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71137::DEBUG::2014-09-25 17:47:59,803::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71137::DEBUG::2014-09-25 17:47:59,804::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71137::DEBUG::2014-09-25 17:47:59,804::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71137::DEBUG::2014-09-25 17:47:59,804::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71137::DEBUG::2014-09-25 17:47:59,805::task::993::Storage.TaskManager.Task::(_decref) Task=`5581a2a5-d95b-41fe-b2f0-194786b0a5c0`::ref 0 aborting False >Thread-71137::DEBUG::2014-09-25 17:47:59,805::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71137::DEBUG::2014-09-25 17:47:59,806::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:47:59,827::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:59,827::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:59,827::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:59,828::__init__::232::IOProcess::(_processLogs) (109442) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:59,828::__init__::232::IOProcess::(_processLogs) (109442) Queuing response >Thread-79::DEBUG::2014-09-25 17:47:59,829::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:47:59,838::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000479146 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:47:59,839::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:59,840::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:59,840::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:59,840::__init__::232::IOProcess::(_processLogs) (109443) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:47:59,841::__init__::232::IOProcess::(_processLogs) (109443) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:59,842::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:59,842::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:59,843::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:59,843::__init__::232::IOProcess::(_processLogs) (109444) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:59,844::__init__::232::IOProcess::(_processLogs) (109444) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:59,844::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:59,844::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:59,845::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:59,845::__init__::232::IOProcess::(_processLogs) (109445) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:59,845::__init__::232::IOProcess::(_processLogs) (109445) Queuing response >Thread-29::DEBUG::2014-09-25 17:47:59,846::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:47:59,847::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:47:59,847::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:47:59,848::__init__::232::IOProcess::(_processLogs) (109446) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:47:59,848::__init__::232::IOProcess::(_processLogs) (109446) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:01,422::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:01,423::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71138::DEBUG::2014-09-25 17:48:01,426::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:01,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:01,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135212 s, 75.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:48:03,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:03,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137325 s, 74.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:04,283::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:04,283::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:04,283::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:04,283::__init__::232::IOProcess::(_processLogs) (109447) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:04,294::__init__::232::IOProcess::(_processLogs) (109447) Queuing response >Thread-81::DEBUG::2014-09-25 17:48:04,296::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:48:04,313::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00322482 s, 115 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:04,314::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:04,314::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:04,315::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:04,315::__init__::232::IOProcess::(_processLogs) (109448) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:04,316::__init__::232::IOProcess::(_processLogs) (109448) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:04,444::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:04,445::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71139::DEBUG::2014-09-25 17:48:04,447::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:05,657::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:05,687::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158819 s, 64.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:06,391::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:06,391::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:06,392::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:06,392::__init__::232::IOProcess::(_processLogs) (109449) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:06,392::__init__::232::IOProcess::(_processLogs) (109449) Queuing response >Thread-80::DEBUG::2014-09-25 17:48:06,394::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:48:06,403::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000554006 s, 682 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:06,404::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:06,404::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:06,405::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:06,405::__init__::232::IOProcess::(_processLogs) (109450) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:06,405::__init__::232::IOProcess::(_processLogs) (109450) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:07,464::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:07,465::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71140::DEBUG::2014-09-25 17:48:07,467::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:07,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:07,729::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135255 s, 75.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:48:09,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:09,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134834 s, 75.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:09,834::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:09,835::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71141::DEBUG::2014-09-25 17:48:09,835::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71141::DEBUG::2014-09-25 17:48:09,837::task::595::Storage.TaskManager.Task::(_updateState) Task=`4302e056-921d-4725-81ec-2426336a0653`::moving from state init -> state preparing >Thread-71141::INFO::2014-09-25 17:48:09,838::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71141::INFO::2014-09-25 17:48:09,838::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71141::DEBUG::2014-09-25 17:48:09,839::task::1191::Storage.TaskManager.Task::(prepare) Task=`4302e056-921d-4725-81ec-2426336a0653`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71141::DEBUG::2014-09-25 17:48:09,839::task::595::Storage.TaskManager.Task::(_updateState) Task=`4302e056-921d-4725-81ec-2426336a0653`::moving from state preparing -> state finished >Thread-71141::DEBUG::2014-09-25 17:48:09,839::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71141::DEBUG::2014-09-25 17:48:09,839::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71141::DEBUG::2014-09-25 17:48:09,839::task::993::Storage.TaskManager.Task::(_decref) Task=`4302e056-921d-4725-81ec-2426336a0653`::ref 0 aborting False >Thread-71141::DEBUG::2014-09-25 17:48:09,840::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71141::DEBUG::2014-09-25 17:48:09,840::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:48:09,848::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:09,848::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:09,848::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:09,849::__init__::232::IOProcess::(_processLogs) (109451) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:09,849::__init__::232::IOProcess::(_processLogs) (109451) Queuing response >Thread-79::DEBUG::2014-09-25 17:48:09,850::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:09,858::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:09,861::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71142::DEBUG::2014-09-25 17:48:09,861::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-79::DEBUG::2014-09-25 17:48:09,861::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000756437 s, 743 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:09,865::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:09,867::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-71142::DEBUG::2014-09-25 17:48:09,867::task::595::Storage.TaskManager.Task::(_updateState) Task=`ee0217fb-927d-4541-bf50-a65a69febcfe`::moving from state init -> state preparing >Thread-29::DEBUG::2014-09-25 17:48:09,868::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-71142::INFO::2014-09-25 17:48:09,868::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-29::DEBUG::2014-09-25 17:48:09,869::__init__::232::IOProcess::(_processLogs) (109452) Got request for method 'statvfs' >Thread-71142::DEBUG::2014-09-25 17:48:09,872::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8f3fd6ef-3f64-4d56-b25e-c168356353c3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-29::DEBUG::2014-09-25 17:48:09,872::__init__::232::IOProcess::(_processLogs) (109452) Queuing response >Thread-71142::DEBUG::2014-09-25 17:48:09,874::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71142::DEBUG::2014-09-25 17:48:09,875::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-29::DEBUG::2014-09-25 17:48:09,876::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-71142::DEBUG::2014-09-25 17:48:09,877::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8f3fd6ef-3f64-4d56-b25e-c168356353c3`::Granted request >Thread-29::DEBUG::2014-09-25 17:48:09,878::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-71142::DEBUG::2014-09-25 17:48:09,880::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ee0217fb-927d-4541-bf50-a65a69febcfe`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-29::DEBUG::2014-09-25 17:48:09,880::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-71142::DEBUG::2014-09-25 17:48:09,881::task::993::Storage.TaskManager.Task::(_decref) Task=`ee0217fb-927d-4541-bf50-a65a69febcfe`::ref 1 aborting False >Thread-29::DEBUG::2014-09-25 17:48:09,881::__init__::232::IOProcess::(_processLogs) (109453) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:09,883::__init__::232::IOProcess::(_processLogs) (109453) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:09,885::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:09,887::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:09,889::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:09,891::__init__::232::IOProcess::(_processLogs) (109454) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:09,893::__init__::232::IOProcess::(_processLogs) (109454) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:09,895::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:09,897::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:09,900::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:09,901::__init__::232::IOProcess::(_processLogs) (109455) Got request for method 'access' >Thread-71142::INFO::2014-09-25 17:48:09,902::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-29::DEBUG::2014-09-25 17:48:09,903::__init__::232::IOProcess::(_processLogs) (109455) Queuing response >Thread-71142::DEBUG::2014-09-25 17:48:09,903::task::1191::Storage.TaskManager.Task::(prepare) Task=`ee0217fb-927d-4541-bf50-a65a69febcfe`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71142::DEBUG::2014-09-25 17:48:09,904::task::595::Storage.TaskManager.Task::(_updateState) Task=`ee0217fb-927d-4541-bf50-a65a69febcfe`::moving from state preparing -> state finished >Thread-71142::DEBUG::2014-09-25 17:48:09,904::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71142::DEBUG::2014-09-25 17:48:09,905::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71142::DEBUG::2014-09-25 17:48:09,905::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71142::DEBUG::2014-09-25 17:48:09,905::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71142::DEBUG::2014-09-25 17:48:09,905::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71142::DEBUG::2014-09-25 17:48:09,906::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71142::DEBUG::2014-09-25 17:48:09,906::task::993::Storage.TaskManager.Task::(_decref) Task=`ee0217fb-927d-4541-bf50-a65a69febcfe`::ref 0 aborting False >Thread-71142::DEBUG::2014-09-25 17:48:09,906::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71142::DEBUG::2014-09-25 17:48:09,907::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:10,482::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:10,484::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71143::DEBUG::2014-09-25 17:48:10,487::task::595::Storage.TaskManager.Task::(_updateState) Task=`d79bfc77-f96d-4ad9-99c3-c1648362708f`::moving from state init -> state preparing >Thread-71143::INFO::2014-09-25 17:48:10,487::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71143::INFO::2014-09-25 17:48:10,488::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000756437', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000554006', 'lastCheck': '4.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00322482', 'lastCheck': '6.2', 'valid': True}} >Thread-71143::DEBUG::2014-09-25 17:48:10,488::task::1191::Storage.TaskManager.Task::(prepare) Task=`d79bfc77-f96d-4ad9-99c3-c1648362708f`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000756437', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000554006', 'lastCheck': '4.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00322482', 'lastCheck': '6.2', 'valid': True}} >Thread-71143::DEBUG::2014-09-25 17:48:10,488::task::595::Storage.TaskManager.Task::(_updateState) Task=`d79bfc77-f96d-4ad9-99c3-c1648362708f`::moving from state preparing -> state finished >Thread-71143::DEBUG::2014-09-25 17:48:10,488::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71143::DEBUG::2014-09-25 17:48:10,489::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71143::DEBUG::2014-09-25 17:48:10,489::task::993::Storage.TaskManager.Task::(_decref) Task=`d79bfc77-f96d-4ad9-99c3-c1648362708f`::ref 0 aborting False >Thread-71143::DEBUG::2014-09-25 17:48:10,491::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:10,517::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:10,518::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71144::DEBUG::2014-09-25 17:48:10,520::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:11,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:11,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136852 s, 74.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:13,578::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:13,579::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71145::DEBUG::2014-09-25 17:48:13,581::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:13,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:13,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159836 s, 64.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:14,318::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:14,319::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:14,319::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:14,319::__init__::232::IOProcess::(_processLogs) (109456) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:14,319::__init__::232::IOProcess::(_processLogs) (109456) Queuing response >Thread-81::DEBUG::2014-09-25 17:48:14,321::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:48:14,330::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0004568 s, 812 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:14,331::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:14,331::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:14,331::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:14,332::__init__::232::IOProcess::(_processLogs) (109457) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:14,332::__init__::232::IOProcess::(_processLogs) (109457) Queuing response >Dummy-148::DEBUG::2014-09-25 17:48:15,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:15,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138697 s, 73.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:16,409::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:16,410::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:16,411::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:16,413::__init__::232::IOProcess::(_processLogs) (109458) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:16,415::__init__::232::IOProcess::(_processLogs) (109458) Queuing response >Thread-80::DEBUG::2014-09-25 17:48:16,416::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:48:16,427::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000527342 s, 717 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:16,430::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:16,432::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:16,433::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:16,434::__init__::232::IOProcess::(_processLogs) (109459) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:16,436::__init__::232::IOProcess::(_processLogs) (109459) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:16,599::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:16,602::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71146::DEBUG::2014-09-25 17:48:16,607::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:17,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:17,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186329 s, 55.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:19,624::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:19,626::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71147::DEBUG::2014-09-25 17:48:19,628::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:48:19,899::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:19,899::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:19,900::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:19,900::__init__::232::IOProcess::(_processLogs) (109460) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:19,909::__init__::232::IOProcess::(_processLogs) (109460) Queuing response >Thread-79::DEBUG::2014-09-25 17:48:19,910::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:48:19,927::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00220384 s, 255 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:19,928::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:19,928::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:19,928::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:19,928::__init__::232::IOProcess::(_processLogs) (109461) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:19,932::__init__::232::IOProcess::(_processLogs) (109461) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:19,933::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:19,933::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:19,934::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:19,934::__init__::232::IOProcess::(_processLogs) (109462) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:19,935::__init__::232::IOProcess::(_processLogs) (109462) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:19,935::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:19,935::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:19,935::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:19,936::__init__::232::IOProcess::(_processLogs) (109463) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:19,936::__init__::232::IOProcess::(_processLogs) (109463) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:19,937::__init__::232::IOProcess::(_processLogs) Receiving request... >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:19,938::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-29::DEBUG::2014-09-25 17:48:19,939::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:19,941::__init__::232::IOProcess::(_processLogs) Extracting request information... >JsonRpcServer::DEBUG::2014-09-25 17:48:19,942::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71148::DEBUG::2014-09-25 17:48:19,942::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-29::DEBUG::2014-09-25 17:48:19,944::__init__::232::IOProcess::(_processLogs) (109464) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:19,946::__init__::232::IOProcess::(_processLogs) (109464) Queuing response >Thread-71148::DEBUG::2014-09-25 17:48:19,948::task::595::Storage.TaskManager.Task::(_updateState) Task=`211537e5-2d63-4256-8c71-7dc2ae8bfe9d`::moving from state init -> state preparing >Thread-71148::INFO::2014-09-25 17:48:19,948::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71148::INFO::2014-09-25 17:48:19,948::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71148::DEBUG::2014-09-25 17:48:19,949::task::1191::Storage.TaskManager.Task::(prepare) Task=`211537e5-2d63-4256-8c71-7dc2ae8bfe9d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71148::DEBUG::2014-09-25 17:48:19,949::task::595::Storage.TaskManager.Task::(_updateState) Task=`211537e5-2d63-4256-8c71-7dc2ae8bfe9d`::moving from state preparing -> state finished >Thread-71148::DEBUG::2014-09-25 17:48:19,949::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71148::DEBUG::2014-09-25 17:48:19,949::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71148::DEBUG::2014-09-25 17:48:19,950::task::993::Storage.TaskManager.Task::(_decref) Task=`211537e5-2d63-4256-8c71-7dc2ae8bfe9d`::ref 0 aborting False >Thread-71148::DEBUG::2014-09-25 17:48:19,950::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71148::DEBUG::2014-09-25 17:48:19,950::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:19,964::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:19,966::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71149::DEBUG::2014-09-25 17:48:19,966::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71149::DEBUG::2014-09-25 17:48:19,967::task::595::Storage.TaskManager.Task::(_updateState) Task=`e902343d-d798-4765-b2e6-b90fd1ba6a01`::moving from state init -> state preparing >Thread-71149::INFO::2014-09-25 17:48:19,968::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71149::DEBUG::2014-09-25 17:48:19,968::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`42c51144-653a-4d82-8066-1d8c89892ab8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71149::DEBUG::2014-09-25 17:48:19,969::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71149::DEBUG::2014-09-25 17:48:19,969::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71149::DEBUG::2014-09-25 17:48:19,969::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`42c51144-653a-4d82-8066-1d8c89892ab8`::Granted request >Thread-71149::DEBUG::2014-09-25 17:48:19,970::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e902343d-d798-4765-b2e6-b90fd1ba6a01`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71149::DEBUG::2014-09-25 17:48:19,970::task::993::Storage.TaskManager.Task::(_decref) Task=`e902343d-d798-4765-b2e6-b90fd1ba6a01`::ref 1 aborting False >Thread-71149::INFO::2014-09-25 17:48:19,974::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71149::DEBUG::2014-09-25 17:48:19,975::task::1191::Storage.TaskManager.Task::(prepare) Task=`e902343d-d798-4765-b2e6-b90fd1ba6a01`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71149::DEBUG::2014-09-25 17:48:19,975::task::595::Storage.TaskManager.Task::(_updateState) Task=`e902343d-d798-4765-b2e6-b90fd1ba6a01`::moving from state preparing -> state finished >Thread-71149::DEBUG::2014-09-25 17:48:19,975::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71149::DEBUG::2014-09-25 17:48:19,975::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71149::DEBUG::2014-09-25 17:48:19,976::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71149::DEBUG::2014-09-25 17:48:19,976::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71149::DEBUG::2014-09-25 17:48:19,976::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71149::DEBUG::2014-09-25 17:48:19,977::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71149::DEBUG::2014-09-25 17:48:19,977::task::993::Storage.TaskManager.Task::(_decref) Task=`e902343d-d798-4765-b2e6-b90fd1ba6a01`::ref 0 aborting False >Thread-71149::DEBUG::2014-09-25 17:48:19,977::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71149::DEBUG::2014-09-25 17:48:19,978::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:19,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:20,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149315 s, 68.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:48:22,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:22,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134252 s, 76.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:22,644::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:22,646::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71150::DEBUG::2014-09-25 17:48:22,648::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:24,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:24,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135807 s, 75.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:24,334::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:24,335::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:24,335::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:24,335::__init__::232::IOProcess::(_processLogs) (109465) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:24,336::__init__::232::IOProcess::(_processLogs) (109465) Queuing response >Thread-81::DEBUG::2014-09-25 17:48:24,337::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:48:24,347::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000367567 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:24,348::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:24,348::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:24,348::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:24,349::__init__::232::IOProcess::(_processLogs) (109466) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:24,349::__init__::232::IOProcess::(_processLogs) (109466) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:25,663::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:25,664::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71151::DEBUG::2014-09-25 17:48:25,667::task::595::Storage.TaskManager.Task::(_updateState) Task=`89e93b57-be94-4310-aa5c-27e2bfa8c042`::moving from state init -> state preparing >Thread-71151::INFO::2014-09-25 17:48:25,667::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71151::INFO::2014-09-25 17:48:25,668::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00220384', 'lastCheck': '5.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000527342', 'lastCheck': '9.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000367567', 'lastCheck': '1.3', 'valid': True}} >Thread-71151::DEBUG::2014-09-25 17:48:25,668::task::1191::Storage.TaskManager.Task::(prepare) Task=`89e93b57-be94-4310-aa5c-27e2bfa8c042`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00220384', 'lastCheck': '5.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000527342', 'lastCheck': '9.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000367567', 'lastCheck': '1.3', 'valid': True}} >Thread-71151::DEBUG::2014-09-25 17:48:25,668::task::595::Storage.TaskManager.Task::(_updateState) Task=`89e93b57-be94-4310-aa5c-27e2bfa8c042`::moving from state preparing -> state finished >Thread-71151::DEBUG::2014-09-25 17:48:25,669::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71151::DEBUG::2014-09-25 17:48:25,669::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71151::DEBUG::2014-09-25 17:48:25,669::task::993::Storage.TaskManager.Task::(_decref) Task=`89e93b57-be94-4310-aa5c-27e2bfa8c042`::ref 0 aborting False >Thread-71151::DEBUG::2014-09-25 17:48:25,671::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:25,693::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:25,694::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71152::DEBUG::2014-09-25 17:48:25,696::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:26,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:26,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134426 s, 76.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:26,439::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:26,439::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:26,440::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:26,440::__init__::232::IOProcess::(_processLogs) (109467) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:26,440::__init__::232::IOProcess::(_processLogs) (109467) Queuing response >Thread-80::DEBUG::2014-09-25 17:48:26,442::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:48:26,451::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000810618 s, 466 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:26,452::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:26,452::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:26,453::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:26,453::__init__::232::IOProcess::(_processLogs) (109468) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:26,454::__init__::232::IOProcess::(_processLogs) (109468) Queuing response >Dummy-148::DEBUG::2014-09-25 17:48:28,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:28,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135637 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:28,755::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:28,756::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71153::DEBUG::2014-09-25 17:48:28,758::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:48:29,944::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:29,944::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:29,945::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:48:29,946::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:48:29,946::__init__::232::IOProcess::(_processLogs) (109469) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:29,955::__init__::232::IOProcess::(_processLogs) (109469) Queuing response >Thread-79::DEBUG::2014-09-25 17:48:29,956::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000367013 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:29,957::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:29,957::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:29,958::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:29,958::__init__::232::IOProcess::(_processLogs) (109470) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:29,958::__init__::232::IOProcess::(_processLogs) (109470) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:29,959::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:29,960::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:29,960::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:29,960::__init__::232::IOProcess::(_processLogs) (109471) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:29,960::__init__::232::IOProcess::(_processLogs) (109471) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:29,961::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:29,961::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:29,962::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:29,963::__init__::232::IOProcess::(_processLogs) (109472) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:29,963::__init__::232::IOProcess::(_processLogs) (109472) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:29,963::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:29,964::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:29,964::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:29,964::__init__::232::IOProcess::(_processLogs) (109473) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:29,964::__init__::232::IOProcess::(_processLogs) (109473) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:30,006::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:30,008::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71154::DEBUG::2014-09-25 17:48:30,008::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71154::DEBUG::2014-09-25 17:48:30,010::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b885dba-d85e-42cb-8a60-c874c0d08ba2`::moving from state init -> state preparing >Thread-71154::INFO::2014-09-25 17:48:30,010::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71154::INFO::2014-09-25 17:48:30,011::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71154::DEBUG::2014-09-25 17:48:30,011::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b885dba-d85e-42cb-8a60-c874c0d08ba2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71154::DEBUG::2014-09-25 17:48:30,011::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b885dba-d85e-42cb-8a60-c874c0d08ba2`::moving from state preparing -> state finished >Thread-71154::DEBUG::2014-09-25 17:48:30,011::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71154::DEBUG::2014-09-25 17:48:30,011::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71154::DEBUG::2014-09-25 17:48:30,012::task::993::Storage.TaskManager.Task::(_decref) Task=`1b885dba-d85e-42cb-8a60-c874c0d08ba2`::ref 0 aborting False >Thread-71154::DEBUG::2014-09-25 17:48:30,012::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71154::DEBUG::2014-09-25 17:48:30,012::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:30,025::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71155::DEBUG::2014-09-25 17:48:30,027::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:48:30,027::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71155::DEBUG::2014-09-25 17:48:30,029::task::595::Storage.TaskManager.Task::(_updateState) Task=`e8142dd2-fce5-4150-97b6-1f89037e3a33`::moving from state init -> state preparing >Thread-71155::INFO::2014-09-25 17:48:30,029::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71155::DEBUG::2014-09-25 17:48:30,030::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`29b90207-0707-4544-a538-15e1e2b3c287`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71155::DEBUG::2014-09-25 17:48:30,030::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71155::DEBUG::2014-09-25 17:48:30,030::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71155::DEBUG::2014-09-25 17:48:30,031::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`29b90207-0707-4544-a538-15e1e2b3c287`::Granted request >Thread-71155::DEBUG::2014-09-25 17:48:30,031::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e8142dd2-fce5-4150-97b6-1f89037e3a33`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71155::DEBUG::2014-09-25 17:48:30,031::task::993::Storage.TaskManager.Task::(_decref) Task=`e8142dd2-fce5-4150-97b6-1f89037e3a33`::ref 1 aborting False >Thread-71155::INFO::2014-09-25 17:48:30,036::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71155::DEBUG::2014-09-25 17:48:30,036::task::1191::Storage.TaskManager.Task::(prepare) Task=`e8142dd2-fce5-4150-97b6-1f89037e3a33`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71155::DEBUG::2014-09-25 17:48:30,036::task::595::Storage.TaskManager.Task::(_updateState) Task=`e8142dd2-fce5-4150-97b6-1f89037e3a33`::moving from state preparing -> state finished >Thread-71155::DEBUG::2014-09-25 17:48:30,036::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71155::DEBUG::2014-09-25 17:48:30,037::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71155::DEBUG::2014-09-25 17:48:30,037::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71155::DEBUG::2014-09-25 17:48:30,037::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71155::DEBUG::2014-09-25 17:48:30,037::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71155::DEBUG::2014-09-25 17:48:30,038::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71155::DEBUG::2014-09-25 17:48:30,038::task::993::Storage.TaskManager.Task::(_decref) Task=`e8142dd2-fce5-4150-97b6-1f89037e3a33`::ref 0 aborting False >Thread-71155::DEBUG::2014-09-25 17:48:30,038::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71155::DEBUG::2014-09-25 17:48:30,039::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:30,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:30,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134741 s, 76.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:31,774::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:31,776::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71156::DEBUG::2014-09-25 17:48:31,778::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:32,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:32,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146449 s, 69.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:48:34,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:34,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151681 s, 67.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:34,352::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:34,352::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:34,352::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:34,352::__init__::232::IOProcess::(_processLogs) (109474) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:34,353::__init__::232::IOProcess::(_processLogs) (109474) Queuing response >Thread-81::DEBUG::2014-09-25 17:48:34,354::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:48:34,362::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000502578 s, 738 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:34,364::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:34,364::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:34,364::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:34,364::__init__::232::IOProcess::(_processLogs) (109475) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:34,365::__init__::232::IOProcess::(_processLogs) (109475) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:34,794::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:34,795::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71157::DEBUG::2014-09-25 17:48:34,797::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:36,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:36,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136745 s, 74.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:36,456::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:36,456::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:36,456::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:36,457::__init__::232::IOProcess::(_processLogs) (109476) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:36,457::__init__::232::IOProcess::(_processLogs) (109476) Queuing response >Thread-80::DEBUG::2014-09-25 17:48:36,459::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:48:36,467::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000422662 s, 894 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:36,468::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:36,468::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:36,469::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:36,469::__init__::232::IOProcess::(_processLogs) (109477) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:36,470::__init__::232::IOProcess::(_processLogs) (109477) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:37,812::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:37,813::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71158::DEBUG::2014-09-25 17:48:37,815::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:38,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:38,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134408 s, 76.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:39,966::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:39,967::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-79::DEBUG::2014-09-25 17:48:39,968::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:48:39,968::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:39,977::__init__::232::IOProcess::(_processLogs) (109478) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:39,977::__init__::232::IOProcess::(_processLogs) (109478) Queuing response >Thread-79::DEBUG::2014-09-25 17:48:39,978::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000406652 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:39,980::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:39,980::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:39,980::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:39,980::__init__::232::IOProcess::(_processLogs) (109479) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:39,981::__init__::232::IOProcess::(_processLogs) (109479) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:39,982::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:39,983::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:39,983::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:39,983::__init__::232::IOProcess::(_processLogs) (109480) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:39,983::__init__::232::IOProcess::(_processLogs) (109480) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:39,984::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:39,985::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:39,985::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:39,986::__init__::232::IOProcess::(_processLogs) (109481) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:39,986::__init__::232::IOProcess::(_processLogs) (109481) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:39,986::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:39,986::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:39,987::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:39,987::__init__::232::IOProcess::(_processLogs) (109482) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:39,987::__init__::232::IOProcess::(_processLogs) (109482) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:40,071::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71159::DEBUG::2014-09-25 17:48:40,072::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:48:40,073::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71159::DEBUG::2014-09-25 17:48:40,075::task::595::Storage.TaskManager.Task::(_updateState) Task=`1894caa8-e2ce-4ec1-bcb0-bcaf930344bb`::moving from state init -> state preparing >Thread-71159::INFO::2014-09-25 17:48:40,075::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71159::INFO::2014-09-25 17:48:40,076::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71159::DEBUG::2014-09-25 17:48:40,076::task::1191::Storage.TaskManager.Task::(prepare) Task=`1894caa8-e2ce-4ec1-bcb0-bcaf930344bb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71159::DEBUG::2014-09-25 17:48:40,076::task::595::Storage.TaskManager.Task::(_updateState) Task=`1894caa8-e2ce-4ec1-bcb0-bcaf930344bb`::moving from state preparing -> state finished >Thread-71159::DEBUG::2014-09-25 17:48:40,077::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71159::DEBUG::2014-09-25 17:48:40,077::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71159::DEBUG::2014-09-25 17:48:40,077::task::993::Storage.TaskManager.Task::(_decref) Task=`1894caa8-e2ce-4ec1-bcb0-bcaf930344bb`::ref 0 aborting False >Thread-71159::DEBUG::2014-09-25 17:48:40,077::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71159::DEBUG::2014-09-25 17:48:40,078::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:40,090::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71160::DEBUG::2014-09-25 17:48:40,092::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:48:40,092::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71160::DEBUG::2014-09-25 17:48:40,094::task::595::Storage.TaskManager.Task::(_updateState) Task=`91b5bc0c-6db5-4333-afb1-b6c48f9cf160`::moving from state init -> state preparing >Thread-71160::INFO::2014-09-25 17:48:40,094::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71160::DEBUG::2014-09-25 17:48:40,095::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b713d0a0-efd8-4706-bcad-2cc949498574`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71160::DEBUG::2014-09-25 17:48:40,095::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71160::DEBUG::2014-09-25 17:48:40,095::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71160::DEBUG::2014-09-25 17:48:40,096::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b713d0a0-efd8-4706-bcad-2cc949498574`::Granted request >Thread-71160::DEBUG::2014-09-25 17:48:40,096::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`91b5bc0c-6db5-4333-afb1-b6c48f9cf160`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71160::DEBUG::2014-09-25 17:48:40,096::task::993::Storage.TaskManager.Task::(_decref) Task=`91b5bc0c-6db5-4333-afb1-b6c48f9cf160`::ref 1 aborting False >Thread-71160::INFO::2014-09-25 17:48:40,101::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71160::DEBUG::2014-09-25 17:48:40,101::task::1191::Storage.TaskManager.Task::(prepare) Task=`91b5bc0c-6db5-4333-afb1-b6c48f9cf160`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71160::DEBUG::2014-09-25 17:48:40,101::task::595::Storage.TaskManager.Task::(_updateState) Task=`91b5bc0c-6db5-4333-afb1-b6c48f9cf160`::moving from state preparing -> state finished >Thread-71160::DEBUG::2014-09-25 17:48:40,102::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71160::DEBUG::2014-09-25 17:48:40,102::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71160::DEBUG::2014-09-25 17:48:40,102::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71160::DEBUG::2014-09-25 17:48:40,102::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71160::DEBUG::2014-09-25 17:48:40,103::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71160::DEBUG::2014-09-25 17:48:40,103::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71160::DEBUG::2014-09-25 17:48:40,103::task::993::Storage.TaskManager.Task::(_decref) Task=`91b5bc0c-6db5-4333-afb1-b6c48f9cf160`::ref 0 aborting False >Thread-71160::DEBUG::2014-09-25 17:48:40,104::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71160::DEBUG::2014-09-25 17:48:40,104::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:40,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:40,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.263653 s, 3.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:40,830::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:40,832::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71161::DEBUG::2014-09-25 17:48:40,834::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b28d59c-9055-49c7-8090-33ba3ebd42ff`::moving from state init -> state preparing >Thread-71161::INFO::2014-09-25 17:48:40,835::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71161::INFO::2014-09-25 17:48:40,835::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000406652', 'lastCheck': '0.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000422662', 'lastCheck': '4.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000502578', 'lastCheck': '6.5', 'valid': True}} >Thread-71161::DEBUG::2014-09-25 17:48:40,835::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b28d59c-9055-49c7-8090-33ba3ebd42ff`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000406652', 'lastCheck': '0.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000422662', 'lastCheck': '4.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000502578', 'lastCheck': '6.5', 'valid': True}} >Thread-71161::DEBUG::2014-09-25 17:48:40,836::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b28d59c-9055-49c7-8090-33ba3ebd42ff`::moving from state preparing -> state finished >Thread-71161::DEBUG::2014-09-25 17:48:40,836::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71161::DEBUG::2014-09-25 17:48:40,836::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71161::DEBUG::2014-09-25 17:48:40,836::task::993::Storage.TaskManager.Task::(_decref) Task=`1b28d59c-9055-49c7-8090-33ba3ebd42ff`::ref 0 aborting False >Thread-71161::DEBUG::2014-09-25 17:48:40,839::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:40,859::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:40,860::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71162::DEBUG::2014-09-25 17:48:40,862::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:42,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:42,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138431 s, 74.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:43,921::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:43,922::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71163::DEBUG::2014-09-25 17:48:43,924::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:48:44,367::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:44,367::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:44,368::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:44,368::__init__::232::IOProcess::(_processLogs) (109483) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:44,368::__init__::232::IOProcess::(_processLogs) (109483) Queuing response >Thread-81::DEBUG::2014-09-25 17:48:44,369::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:48:44,378::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000508068 s, 730 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:44,380::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:44,380::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:44,380::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:44,380::__init__::232::IOProcess::(_processLogs) (109484) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:44,381::__init__::232::IOProcess::(_processLogs) (109484) Queuing response >Dummy-148::DEBUG::2014-09-25 17:48:44,757::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:44,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134102 s, 76.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:46,473::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:46,475::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:46,477::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:48:46,478::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:48:46,480::__init__::232::IOProcess::(_processLogs) (109485) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:46,489::__init__::232::IOProcess::(_processLogs) (109485) Queuing response >Thread-80::DEBUG::2014-09-25 17:48:46,491::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000476023 s, 794 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:46,494::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:46,495::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:46,496::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:46,497::__init__::232::IOProcess::(_processLogs) (109486) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:46,499::__init__::232::IOProcess::(_processLogs) (109486) Queuing response >Dummy-148::DEBUG::2014-09-25 17:48:46,799::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:46,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0318545 s, 32.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:46,941::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:46,945::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71164::DEBUG::2014-09-25 17:48:46,953::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:49,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:49,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134533 s, 76.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:49,973::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:49,974::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71165::DEBUG::2014-09-25 17:48:49,977::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-79::DEBUG::2014-09-25 17:48:49,988::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:48:49,992::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:49,992::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:49,993::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:49,993::__init__::232::IOProcess::(_processLogs) (109487) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:48:49,993::__init__::232::IOProcess::(_processLogs) (109487) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:49,995::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:49,995::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:49,995::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:49,995::__init__::232::IOProcess::(_processLogs) (109488) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:48:49,996::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:49,996::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:49,996::__init__::232::IOProcess::(_processLogs) (109488) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:49,996::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:49,997::__init__::232::IOProcess::(_processLogs) (109489) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:48:49,997::__init__::232::IOProcess::(_processLogs) (109489) Queuing response >Thread-79::DEBUG::2014-09-25 17:48:50,002::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:48:50,002::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:50,003::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:50,003::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:50,003::__init__::232::IOProcess::(_processLogs) (109490) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:48:50,027::__init__::232::IOProcess::(_processLogs) (109490) Queuing response >Thread-79::DEBUG::2014-09-25 17:48:50,027::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:48:50,028::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:50,028::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:50,029::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:50,029::__init__::232::IOProcess::(_processLogs) (109491) Got request for method 'readfile' >Thread-79::DEBUG::2014-09-25 17:48:50,030::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:48:50,029::__init__::232::IOProcess::(_processLogs) (109491) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:50,032::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:50,032::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:50,033::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:50,033::__init__::232::IOProcess::(_processLogs) (109492) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:50,033::__init__::232::IOProcess::(_processLogs) (109492) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:50,034::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:50,034::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:50,034::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:50,034::__init__::232::IOProcess::(_processLogs) (109493) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:48:50,035::__init__::232::IOProcess::(_processLogs) (109493) Queuing response >Thread-79::DEBUG::2014-09-25 17:48:50,036::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 17:48:50,036::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:48:50,036::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:48:50,037::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:50,037::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:50,037::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:50,038::__init__::232::IOProcess::(_processLogs) (109494) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:50,039::__init__::232::IOProcess::(_processLogs) (109494) Queuing response >Thread-79::DEBUG::2014-09-25 17:48:50,039::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:48:50,048::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000440705 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:50,049::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:50,049::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:50,050::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:50,050::__init__::232::IOProcess::(_processLogs) (109495) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:50,051::__init__::232::IOProcess::(_processLogs) (109495) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:50,052::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:50,052::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:50,053::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:50,053::__init__::232::IOProcess::(_processLogs) (109496) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:50,053::__init__::232::IOProcess::(_processLogs) (109496) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:50,054::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:50,054::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:50,054::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:50,055::__init__::232::IOProcess::(_processLogs) (109497) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:50,055::__init__::232::IOProcess::(_processLogs) (109497) Queuing response >Thread-29::DEBUG::2014-09-25 17:48:50,056::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:50,056::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:50,056::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:50,057::__init__::232::IOProcess::(_processLogs) (109498) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:48:50,057::__init__::232::IOProcess::(_processLogs) (109498) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:50,134::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:50,135::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71169::DEBUG::2014-09-25 17:48:50,135::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71169::DEBUG::2014-09-25 17:48:50,137::task::595::Storage.TaskManager.Task::(_updateState) Task=`06cfa098-9eb3-41e2-bc68-b0c4591f84c4`::moving from state init -> state preparing >Thread-71169::INFO::2014-09-25 17:48:50,138::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71169::INFO::2014-09-25 17:48:50,138::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71169::DEBUG::2014-09-25 17:48:50,139::task::1191::Storage.TaskManager.Task::(prepare) Task=`06cfa098-9eb3-41e2-bc68-b0c4591f84c4`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71169::DEBUG::2014-09-25 17:48:50,139::task::595::Storage.TaskManager.Task::(_updateState) Task=`06cfa098-9eb3-41e2-bc68-b0c4591f84c4`::moving from state preparing -> state finished >Thread-71169::DEBUG::2014-09-25 17:48:50,139::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71169::DEBUG::2014-09-25 17:48:50,139::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71169::DEBUG::2014-09-25 17:48:50,139::task::993::Storage.TaskManager.Task::(_decref) Task=`06cfa098-9eb3-41e2-bc68-b0c4591f84c4`::ref 0 aborting False >Thread-71169::DEBUG::2014-09-25 17:48:50,140::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71169::DEBUG::2014-09-25 17:48:50,140::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:50,154::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71170::DEBUG::2014-09-25 17:48:50,155::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:48:50,155::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71170::DEBUG::2014-09-25 17:48:50,157::task::595::Storage.TaskManager.Task::(_updateState) Task=`62ae2e87-2b31-43e8-b68f-635106c45ce2`::moving from state init -> state preparing >Thread-71170::INFO::2014-09-25 17:48:50,157::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71170::DEBUG::2014-09-25 17:48:50,159::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1690e928-2ee3-456d-9cdf-91de48a88a0f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71170::DEBUG::2014-09-25 17:48:50,160::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71170::DEBUG::2014-09-25 17:48:50,160::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71170::DEBUG::2014-09-25 17:48:50,160::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1690e928-2ee3-456d-9cdf-91de48a88a0f`::Granted request >Thread-71170::DEBUG::2014-09-25 17:48:50,161::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`62ae2e87-2b31-43e8-b68f-635106c45ce2`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71170::DEBUG::2014-09-25 17:48:50,161::task::993::Storage.TaskManager.Task::(_decref) Task=`62ae2e87-2b31-43e8-b68f-635106c45ce2`::ref 1 aborting False >Thread-71170::INFO::2014-09-25 17:48:50,165::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71170::DEBUG::2014-09-25 17:48:50,166::task::1191::Storage.TaskManager.Task::(prepare) Task=`62ae2e87-2b31-43e8-b68f-635106c45ce2`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71170::DEBUG::2014-09-25 17:48:50,166::task::595::Storage.TaskManager.Task::(_updateState) Task=`62ae2e87-2b31-43e8-b68f-635106c45ce2`::moving from state preparing -> state finished >Thread-71170::DEBUG::2014-09-25 17:48:50,166::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71170::DEBUG::2014-09-25 17:48:50,166::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71170::DEBUG::2014-09-25 17:48:50,167::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71170::DEBUG::2014-09-25 17:48:50,167::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71170::DEBUG::2014-09-25 17:48:50,167::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71170::DEBUG::2014-09-25 17:48:50,167::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71170::DEBUG::2014-09-25 17:48:50,168::task::993::Storage.TaskManager.Task::(_decref) Task=`62ae2e87-2b31-43e8-b68f-635106c45ce2`::ref 0 aborting False >Thread-71170::DEBUG::2014-09-25 17:48:50,168::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71170::DEBUG::2014-09-25 17:48:50,169::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:51,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:51,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131844 s, 77.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:52,993::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:52,994::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71171::DEBUG::2014-09-25 17:48:52,996::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:53,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:53,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131092 s, 78.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:54,383::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:54,384::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:54,384::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:54,384::__init__::232::IOProcess::(_processLogs) (109499) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:54,384::__init__::232::IOProcess::(_processLogs) (109499) Queuing response >Thread-81::DEBUG::2014-09-25 17:48:54,386::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:48:54,395::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000462146 s, 803 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:54,396::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:54,396::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:54,397::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:54,398::__init__::232::IOProcess::(_processLogs) (109500) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:54,398::__init__::232::IOProcess::(_processLogs) (109500) Queuing response >Dummy-148::DEBUG::2014-09-25 17:48:55,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:55,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131371 s, 77.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:56,011::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:56,012::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71172::DEBUG::2014-09-25 17:48:56,015::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f5a8f7f-d834-4be0-be43-aef54f1a80cc`::moving from state init -> state preparing >Thread-71172::INFO::2014-09-25 17:48:56,016::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71172::INFO::2014-09-25 17:48:56,016::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000440705', 'lastCheck': '6.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000476023', 'lastCheck': '9.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000462146', 'lastCheck': '1.6', 'valid': True}} >Thread-71172::DEBUG::2014-09-25 17:48:56,016::task::1191::Storage.TaskManager.Task::(prepare) Task=`0f5a8f7f-d834-4be0-be43-aef54f1a80cc`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000440705', 'lastCheck': '6.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000476023', 'lastCheck': '9.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000462146', 'lastCheck': '1.6', 'valid': True}} >Thread-71172::DEBUG::2014-09-25 17:48:56,017::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f5a8f7f-d834-4be0-be43-aef54f1a80cc`::moving from state preparing -> state finished >Thread-71172::DEBUG::2014-09-25 17:48:56,017::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71172::DEBUG::2014-09-25 17:48:56,017::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71172::DEBUG::2014-09-25 17:48:56,017::task::993::Storage.TaskManager.Task::(_decref) Task=`0f5a8f7f-d834-4be0-be43-aef54f1a80cc`::ref 0 aborting False >Thread-71172::DEBUG::2014-09-25 17:48:56,020::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:56,042::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:56,043::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71173::DEBUG::2014-09-25 17:48:56,045::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:48:56,501::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:56,502::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:56,502::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:56,502::__init__::232::IOProcess::(_processLogs) (109501) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:56,502::__init__::232::IOProcess::(_processLogs) (109501) Queuing response >Thread-80::DEBUG::2014-09-25 17:48:56,504::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:48:56,513::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000485436 s, 779 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:48:56,514::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:48:56,514::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:48:56,514::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:48:56,514::__init__::232::IOProcess::(_processLogs) (109502) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:48:56,515::__init__::232::IOProcess::(_processLogs) (109502) Queuing response >Dummy-148::DEBUG::2014-09-25 17:48:57,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:57,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204221 s, 50.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:48:59,106::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:48:59,107::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71174::DEBUG::2014-09-25 17:48:59,109::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:48:59,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:48:59,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0139407 s, 73.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:00,058::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:00,058::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:00,059::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:00,059::__init__::232::IOProcess::(_processLogs) (109503) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:00,059::__init__::232::IOProcess::(_processLogs) (109503) Queuing response >Thread-79::DEBUG::2014-09-25 17:49:00,060::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:49:00,069::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000512941 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:00,070::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:00,071::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:00,071::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:00,073::__init__::232::IOProcess::(_processLogs) (109504) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:00,073::__init__::232::IOProcess::(_processLogs) (109504) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:00,073::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:00,073::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:00,074::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:00,075::__init__::232::IOProcess::(_processLogs) (109505) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:00,075::__init__::232::IOProcess::(_processLogs) (109505) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:00,075::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:00,076::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:00,076::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:00,077::__init__::232::IOProcess::(_processLogs) (109506) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:00,077::__init__::232::IOProcess::(_processLogs) (109506) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:00,077::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:00,078::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:00,078::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:00,079::__init__::232::IOProcess::(_processLogs) (109507) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:00,079::__init__::232::IOProcess::(_processLogs) (109507) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:00,200::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:00,202::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71175::DEBUG::2014-09-25 17:49:00,202::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71175::DEBUG::2014-09-25 17:49:00,204::task::595::Storage.TaskManager.Task::(_updateState) Task=`e57a7c92-f95b-4fc6-96c2-5206978b4ecf`::moving from state init -> state preparing >Thread-71175::INFO::2014-09-25 17:49:00,204::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71175::INFO::2014-09-25 17:49:00,204::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71175::DEBUG::2014-09-25 17:49:00,205::task::1191::Storage.TaskManager.Task::(prepare) Task=`e57a7c92-f95b-4fc6-96c2-5206978b4ecf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71175::DEBUG::2014-09-25 17:49:00,205::task::595::Storage.TaskManager.Task::(_updateState) Task=`e57a7c92-f95b-4fc6-96c2-5206978b4ecf`::moving from state preparing -> state finished >Thread-71175::DEBUG::2014-09-25 17:49:00,205::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71175::DEBUG::2014-09-25 17:49:00,205::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71175::DEBUG::2014-09-25 17:49:00,206::task::993::Storage.TaskManager.Task::(_decref) Task=`e57a7c92-f95b-4fc6-96c2-5206978b4ecf`::ref 0 aborting False >Thread-71175::DEBUG::2014-09-25 17:49:00,206::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71175::DEBUG::2014-09-25 17:49:00,206::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:00,220::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:00,221::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71176::DEBUG::2014-09-25 17:49:00,222::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71176::DEBUG::2014-09-25 17:49:00,223::task::595::Storage.TaskManager.Task::(_updateState) Task=`ba4b8e5f-528b-4a89-b37f-e5e7141635ac`::moving from state init -> state preparing >Thread-71176::INFO::2014-09-25 17:49:00,223::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71176::DEBUG::2014-09-25 17:49:00,224::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2348d95c-cbad-4a91-a6f0-7d1a21f6cdbf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71176::DEBUG::2014-09-25 17:49:00,224::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71176::DEBUG::2014-09-25 17:49:00,225::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71176::DEBUG::2014-09-25 17:49:00,225::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2348d95c-cbad-4a91-a6f0-7d1a21f6cdbf`::Granted request >Thread-71176::DEBUG::2014-09-25 17:49:00,226::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ba4b8e5f-528b-4a89-b37f-e5e7141635ac`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71176::DEBUG::2014-09-25 17:49:00,226::task::993::Storage.TaskManager.Task::(_decref) Task=`ba4b8e5f-528b-4a89-b37f-e5e7141635ac`::ref 1 aborting False >Thread-71176::INFO::2014-09-25 17:49:00,230::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71176::DEBUG::2014-09-25 17:49:00,230::task::1191::Storage.TaskManager.Task::(prepare) Task=`ba4b8e5f-528b-4a89-b37f-e5e7141635ac`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71176::DEBUG::2014-09-25 17:49:00,231::task::595::Storage.TaskManager.Task::(_updateState) Task=`ba4b8e5f-528b-4a89-b37f-e5e7141635ac`::moving from state preparing -> state finished >Thread-71176::DEBUG::2014-09-25 17:49:00,231::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71176::DEBUG::2014-09-25 17:49:00,231::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71176::DEBUG::2014-09-25 17:49:00,231::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71176::DEBUG::2014-09-25 17:49:00,232::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71176::DEBUG::2014-09-25 17:49:00,232::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71176::DEBUG::2014-09-25 17:49:00,232::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71176::DEBUG::2014-09-25 17:49:00,233::task::993::Storage.TaskManager.Task::(_decref) Task=`ba4b8e5f-528b-4a89-b37f-e5e7141635ac`::ref 0 aborting False >Thread-71176::DEBUG::2014-09-25 17:49:00,233::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71176::DEBUG::2014-09-25 17:49:00,234::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:01,282::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:01,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136224 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:02,124::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:02,126::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71177::DEBUG::2014-09-25 17:49:02,128::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:03,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:03,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131746 s, 77.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:04,399::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:04,399::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:04,399::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:04,400::__init__::232::IOProcess::(_processLogs) (109508) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:04,442::__init__::232::IOProcess::(_processLogs) (109508) Queuing response >Thread-81::DEBUG::2014-09-25 17:49:04,443::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:49:04,463::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00322673 s, 115 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:04,465::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:04,465::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:04,465::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:04,465::__init__::232::IOProcess::(_processLogs) (109509) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:04,466::__init__::232::IOProcess::(_processLogs) (109509) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:05,146::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:05,147::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71178::DEBUG::2014-09-25 17:49:05,149::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:05,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:05,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131796 s, 77.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:06,517::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:06,518::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:06,518::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:06,518::__init__::232::IOProcess::(_processLogs) (109510) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:06,518::__init__::232::IOProcess::(_processLogs) (109510) Queuing response >Thread-80::DEBUG::2014-09-25 17:49:06,520::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:49:06,529::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000474262 s, 797 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:06,530::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:06,531::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:06,531::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:06,533::__init__::232::IOProcess::(_processLogs) (109511) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:06,533::__init__::232::IOProcess::(_processLogs) (109511) Queuing response >Dummy-148::DEBUG::2014-09-25 17:49:07,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:07,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135867 s, 75.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:08,167::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:08,168::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71179::DEBUG::2014-09-25 17:49:08,170::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:09,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:09,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167478 s, 61.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:10,080::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:10,080::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:10,082::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:49:10,082::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:49:10,083::__init__::232::IOProcess::(_processLogs) (109512) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:10,091::__init__::232::IOProcess::(_processLogs) (109512) Queuing response >Thread-79::DEBUG::2014-09-25 17:49:10,092::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000378434 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:10,093::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:10,094::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:10,094::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:10,094::__init__::232::IOProcess::(_processLogs) (109513) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:10,095::__init__::232::IOProcess::(_processLogs) (109513) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:10,096::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:10,097::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:10,097::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:10,097::__init__::232::IOProcess::(_processLogs) (109514) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:10,098::__init__::232::IOProcess::(_processLogs) (109514) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:10,098::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:10,099::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:10,099::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:10,100::__init__::232::IOProcess::(_processLogs) (109515) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:10,100::__init__::232::IOProcess::(_processLogs) (109515) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:10,100::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:10,101::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:10,101::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:10,102::__init__::232::IOProcess::(_processLogs) (109516) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:10,102::__init__::232::IOProcess::(_processLogs) (109516) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:10,262::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71180::DEBUG::2014-09-25 17:49:10,263::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:49:10,263::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71180::DEBUG::2014-09-25 17:49:10,265::task::595::Storage.TaskManager.Task::(_updateState) Task=`edb57a59-314e-4f93-9aae-645818a71842`::moving from state init -> state preparing >Thread-71180::INFO::2014-09-25 17:49:10,266::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71180::INFO::2014-09-25 17:49:10,266::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71180::DEBUG::2014-09-25 17:49:10,267::task::1191::Storage.TaskManager.Task::(prepare) Task=`edb57a59-314e-4f93-9aae-645818a71842`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71180::DEBUG::2014-09-25 17:49:10,267::task::595::Storage.TaskManager.Task::(_updateState) Task=`edb57a59-314e-4f93-9aae-645818a71842`::moving from state preparing -> state finished >Thread-71180::DEBUG::2014-09-25 17:49:10,267::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71180::DEBUG::2014-09-25 17:49:10,267::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71180::DEBUG::2014-09-25 17:49:10,267::task::993::Storage.TaskManager.Task::(_decref) Task=`edb57a59-314e-4f93-9aae-645818a71842`::ref 0 aborting False >Thread-71180::DEBUG::2014-09-25 17:49:10,268::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71180::DEBUG::2014-09-25 17:49:10,268::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:10,283::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:10,284::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71181::DEBUG::2014-09-25 17:49:10,284::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71181::DEBUG::2014-09-25 17:49:10,286::task::595::Storage.TaskManager.Task::(_updateState) Task=`9c4381bf-9d2d-4976-aced-f3ab9760f5ee`::moving from state init -> state preparing >Thread-71181::INFO::2014-09-25 17:49:10,286::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71181::DEBUG::2014-09-25 17:49:10,287::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f8c68fa2-6013-4358-afdd-004328834a3f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71181::DEBUG::2014-09-25 17:49:10,287::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71181::DEBUG::2014-09-25 17:49:10,288::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71181::DEBUG::2014-09-25 17:49:10,288::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f8c68fa2-6013-4358-afdd-004328834a3f`::Granted request >Thread-71181::DEBUG::2014-09-25 17:49:10,289::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9c4381bf-9d2d-4976-aced-f3ab9760f5ee`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71181::DEBUG::2014-09-25 17:49:10,289::task::993::Storage.TaskManager.Task::(_decref) Task=`9c4381bf-9d2d-4976-aced-f3ab9760f5ee`::ref 1 aborting False >Thread-71181::INFO::2014-09-25 17:49:10,293::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71181::DEBUG::2014-09-25 17:49:10,293::task::1191::Storage.TaskManager.Task::(prepare) Task=`9c4381bf-9d2d-4976-aced-f3ab9760f5ee`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71181::DEBUG::2014-09-25 17:49:10,294::task::595::Storage.TaskManager.Task::(_updateState) Task=`9c4381bf-9d2d-4976-aced-f3ab9760f5ee`::moving from state preparing -> state finished >Thread-71181::DEBUG::2014-09-25 17:49:10,294::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71181::DEBUG::2014-09-25 17:49:10,294::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71181::DEBUG::2014-09-25 17:49:10,294::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71181::DEBUG::2014-09-25 17:49:10,295::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71181::DEBUG::2014-09-25 17:49:10,295::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71181::DEBUG::2014-09-25 17:49:10,295::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71181::DEBUG::2014-09-25 17:49:10,296::task::993::Storage.TaskManager.Task::(_decref) Task=`9c4381bf-9d2d-4976-aced-f3ab9760f5ee`::ref 0 aborting False >Thread-71181::DEBUG::2014-09-25 17:49:10,296::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71181::DEBUG::2014-09-25 17:49:10,296::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:11,185::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:11,187::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71182::DEBUG::2014-09-25 17:49:11,189::task::595::Storage.TaskManager.Task::(_updateState) Task=`82cd36ff-deb9-42b7-894f-511c8bd8e44e`::moving from state init -> state preparing >Thread-71182::INFO::2014-09-25 17:49:11,190::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71182::INFO::2014-09-25 17:49:11,190::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000378434', 'lastCheck': '1.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000474262', 'lastCheck': '4.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00322673', 'lastCheck': '6.7', 'valid': True}} >Thread-71182::DEBUG::2014-09-25 17:49:11,190::task::1191::Storage.TaskManager.Task::(prepare) Task=`82cd36ff-deb9-42b7-894f-511c8bd8e44e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000378434', 'lastCheck': '1.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000474262', 'lastCheck': '4.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00322673', 'lastCheck': '6.7', 'valid': True}} >Thread-71182::DEBUG::2014-09-25 17:49:11,191::task::595::Storage.TaskManager.Task::(_updateState) Task=`82cd36ff-deb9-42b7-894f-511c8bd8e44e`::moving from state preparing -> state finished >Thread-71182::DEBUG::2014-09-25 17:49:11,191::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71182::DEBUG::2014-09-25 17:49:11,191::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71182::DEBUG::2014-09-25 17:49:11,191::task::993::Storage.TaskManager.Task::(_decref) Task=`82cd36ff-deb9-42b7-894f-511c8bd8e44e`::ref 0 aborting False >Thread-71182::DEBUG::2014-09-25 17:49:11,194::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:11,218::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:11,219::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71183::DEBUG::2014-09-25 17:49:11,221::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:11,497::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:11,519::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.01357 s, 75.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:49:13,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:13,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136417 s, 75.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:14,281::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:14,282::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71184::DEBUG::2014-09-25 17:49:14,284::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:49:14,468::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:14,468::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:14,469::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:14,469::__init__::232::IOProcess::(_processLogs) (109517) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:14,469::__init__::232::IOProcess::(_processLogs) (109517) Queuing response >Thread-81::DEBUG::2014-09-25 17:49:14,471::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:49:14,479::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000326309 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:14,480::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:14,481::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:14,482::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:14,483::__init__::232::IOProcess::(_processLogs) (109518) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:14,483::__init__::232::IOProcess::(_processLogs) (109518) Queuing response >Dummy-148::DEBUG::2014-09-25 17:49:15,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:15,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134681 s, 76.0 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 17:49:16,533::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:49:16,539::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:16,542::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:16,543::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:16,544::__init__::232::IOProcess::(_processLogs) (109519) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:49:16,546::__init__::232::IOProcess::(_processLogs) (109519) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:16,549::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:16,550::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:16,551::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:16,552::__init__::232::IOProcess::(_processLogs) (109520) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:49:16,554::__init__::232::IOProcess::(_processLogs) (109520) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:16,556::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:16,559::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:16,562::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:16,565::__init__::232::IOProcess::(_processLogs) (109521) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:49:16,567::__init__::232::IOProcess::(_processLogs) (109521) Queuing response >Thread-80::DEBUG::2014-09-25 17:49:16,574::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:49:16,576::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:16,577::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:16,578::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:16,579::__init__::232::IOProcess::(_processLogs) (109522) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:49:16,580::__init__::232::IOProcess::(_processLogs) (109522) Queuing response >Thread-80::DEBUG::2014-09-25 17:49:16,582::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:49:16,584::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:16,585::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:16,586::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:16,588::__init__::232::IOProcess::(_processLogs) (109523) Got request for method 'readfile' >Thread-80::DEBUG::2014-09-25 17:49:16,590::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:49:16,589::__init__::232::IOProcess::(_processLogs) (109523) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:16,594::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:16,595::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:16,596::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:16,597::__init__::232::IOProcess::(_processLogs) (109524) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:16,598::__init__::232::IOProcess::(_processLogs) (109524) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:16,601::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:16,602::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-80::DEBUG::2014-09-25 17:49:16,603::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:49:16,603::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::INFO::2014-09-25 17:49:16,605::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:49:16,606::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:49:16,606::__init__::232::IOProcess::(_processLogs) (109525) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:49:16,609::__init__::232::IOProcess::(_processLogs) (109525) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:16,610::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:16,611::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:16,612::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:16,614::__init__::232::IOProcess::(_processLogs) (109526) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:16,615::__init__::232::IOProcess::(_processLogs) (109526) Queuing response >Thread-80::DEBUG::2014-09-25 17:49:16,617::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:49:16,628::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000465449 s, 812 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:16,631::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:16,632::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:16,634::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:16,636::__init__::232::IOProcess::(_processLogs) (109527) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:16,637::__init__::232::IOProcess::(_processLogs) (109527) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:17,303::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:17,304::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71188::DEBUG::2014-09-25 17:49:17,306::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:17,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:17,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133227 s, 76.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:49:19,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:19,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129552 s, 79.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:20,103::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:20,103::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:20,104::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:20,104::__init__::232::IOProcess::(_processLogs) (109528) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:20,105::__init__::232::IOProcess::(_processLogs) (109528) Queuing response >Thread-79::DEBUG::2014-09-25 17:49:20,105::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:49:20,115::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000397825 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:20,116::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:20,116::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:20,116::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:20,117::__init__::232::IOProcess::(_processLogs) (109529) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:20,117::__init__::232::IOProcess::(_processLogs) (109529) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:20,119::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:20,119::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:20,119::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:20,119::__init__::232::IOProcess::(_processLogs) (109530) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:20,120::__init__::232::IOProcess::(_processLogs) (109530) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:20,121::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:20,121::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:20,121::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:20,121::__init__::232::IOProcess::(_processLogs) (109531) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:20,121::__init__::232::IOProcess::(_processLogs) (109531) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:20,122::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:20,123::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:20,123::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:20,124::__init__::232::IOProcess::(_processLogs) (109532) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:20,124::__init__::232::IOProcess::(_processLogs) (109532) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:20,323::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:20,324::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71189::DEBUG::2014-09-25 17:49:20,326::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:20,327::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:20,329::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71190::DEBUG::2014-09-25 17:49:20,329::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71190::DEBUG::2014-09-25 17:49:20,331::task::595::Storage.TaskManager.Task::(_updateState) Task=`90f78fb3-3a7b-4676-9d95-eadedf563eed`::moving from state init -> state preparing >Thread-71190::INFO::2014-09-25 17:49:20,331::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71190::INFO::2014-09-25 17:49:20,331::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71190::DEBUG::2014-09-25 17:49:20,332::task::1191::Storage.TaskManager.Task::(prepare) Task=`90f78fb3-3a7b-4676-9d95-eadedf563eed`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71190::DEBUG::2014-09-25 17:49:20,332::task::595::Storage.TaskManager.Task::(_updateState) Task=`90f78fb3-3a7b-4676-9d95-eadedf563eed`::moving from state preparing -> state finished >Thread-71190::DEBUG::2014-09-25 17:49:20,332::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71190::DEBUG::2014-09-25 17:49:20,332::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71190::DEBUG::2014-09-25 17:49:20,333::task::993::Storage.TaskManager.Task::(_decref) Task=`90f78fb3-3a7b-4676-9d95-eadedf563eed`::ref 0 aborting False >Thread-71190::DEBUG::2014-09-25 17:49:20,333::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71190::DEBUG::2014-09-25 17:49:20,333::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:20,347::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:20,348::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71191::DEBUG::2014-09-25 17:49:20,348::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71191::DEBUG::2014-09-25 17:49:20,350::task::595::Storage.TaskManager.Task::(_updateState) Task=`edfae12b-aa03-4d7f-a1cb-77fc2a79f91c`::moving from state init -> state preparing >Thread-71191::INFO::2014-09-25 17:49:20,350::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71191::DEBUG::2014-09-25 17:49:20,351::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`481bb4dc-36c5-45e9-a547-7032f93d56f0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71191::DEBUG::2014-09-25 17:49:20,351::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71191::DEBUG::2014-09-25 17:49:20,351::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71191::DEBUG::2014-09-25 17:49:20,352::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`481bb4dc-36c5-45e9-a547-7032f93d56f0`::Granted request >Thread-71191::DEBUG::2014-09-25 17:49:20,352::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`edfae12b-aa03-4d7f-a1cb-77fc2a79f91c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71191::DEBUG::2014-09-25 17:49:20,352::task::993::Storage.TaskManager.Task::(_decref) Task=`edfae12b-aa03-4d7f-a1cb-77fc2a79f91c`::ref 1 aborting False >Thread-71191::INFO::2014-09-25 17:49:20,357::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71191::DEBUG::2014-09-25 17:49:20,357::task::1191::Storage.TaskManager.Task::(prepare) Task=`edfae12b-aa03-4d7f-a1cb-77fc2a79f91c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71191::DEBUG::2014-09-25 17:49:20,357::task::595::Storage.TaskManager.Task::(_updateState) Task=`edfae12b-aa03-4d7f-a1cb-77fc2a79f91c`::moving from state preparing -> state finished >Thread-71191::DEBUG::2014-09-25 17:49:20,357::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71191::DEBUG::2014-09-25 17:49:20,358::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71191::DEBUG::2014-09-25 17:49:20,358::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71191::DEBUG::2014-09-25 17:49:20,358::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71191::DEBUG::2014-09-25 17:49:20,359::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71191::DEBUG::2014-09-25 17:49:20,359::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71191::DEBUG::2014-09-25 17:49:20,359::task::993::Storage.TaskManager.Task::(_decref) Task=`edfae12b-aa03-4d7f-a1cb-77fc2a79f91c`::ref 0 aborting False >Thread-71191::DEBUG::2014-09-25 17:49:20,360::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71191::DEBUG::2014-09-25 17:49:20,360::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:21,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:21,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168672 s, 60.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:23,341::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:23,343::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71192::DEBUG::2014-09-25 17:49:23,345::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:23,754::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:23,783::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188564 s, 54.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:24,484::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:24,484::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:24,484::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:24,485::__init__::232::IOProcess::(_processLogs) (109533) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:24,487::__init__::232::IOProcess::(_processLogs) (109533) Queuing response >Thread-81::DEBUG::2014-09-25 17:49:24,488::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:49:24,507::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00486299 s, 76.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:24,509::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:24,509::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:24,509::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:24,509::__init__::232::IOProcess::(_processLogs) (109534) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:24,510::__init__::232::IOProcess::(_processLogs) (109534) Queuing response >Dummy-148::DEBUG::2014-09-25 17:49:25,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:25,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135453 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:26,359::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:26,360::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71193::DEBUG::2014-09-25 17:49:26,363::task::595::Storage.TaskManager.Task::(_updateState) Task=`ebd472cd-5db1-42cb-8167-ad340013926e`::moving from state init -> state preparing >Thread-71193::INFO::2014-09-25 17:49:26,363::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71193::INFO::2014-09-25 17:49:26,364::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000397825', 'lastCheck': '6.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000465449', 'lastCheck': '9.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00486299', 'lastCheck': '1.9', 'valid': True}} >Thread-71193::DEBUG::2014-09-25 17:49:26,364::task::1191::Storage.TaskManager.Task::(prepare) Task=`ebd472cd-5db1-42cb-8167-ad340013926e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000397825', 'lastCheck': '6.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000465449', 'lastCheck': '9.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00486299', 'lastCheck': '1.9', 'valid': True}} >Thread-71193::DEBUG::2014-09-25 17:49:26,364::task::595::Storage.TaskManager.Task::(_updateState) Task=`ebd472cd-5db1-42cb-8167-ad340013926e`::moving from state preparing -> state finished >Thread-71193::DEBUG::2014-09-25 17:49:26,364::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71193::DEBUG::2014-09-25 17:49:26,365::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71193::DEBUG::2014-09-25 17:49:26,365::task::993::Storage.TaskManager.Task::(_decref) Task=`ebd472cd-5db1-42cb-8167-ad340013926e`::ref 0 aborting False >Thread-71193::DEBUG::2014-09-25 17:49:26,367::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:26,387::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:26,388::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71194::DEBUG::2014-09-25 17:49:26,390::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:49:26,636::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:26,637::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:26,637::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:26,637::__init__::232::IOProcess::(_processLogs) (109535) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:26,638::__init__::232::IOProcess::(_processLogs) (109535) Queuing response >Thread-80::DEBUG::2014-09-25 17:49:26,639::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:49:26,648::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000505382 s, 748 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:26,649::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:26,649::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:26,650::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:26,651::__init__::232::IOProcess::(_processLogs) (109536) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:26,652::__init__::232::IOProcess::(_processLogs) (109536) Queuing response >Dummy-148::DEBUG::2014-09-25 17:49:27,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:27,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135475 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:29,449::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:29,450::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71195::DEBUG::2014-09-25 17:49:29,452::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:29,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:29,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135228 s, 75.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:30,125::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:30,125::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:30,126::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:49:30,127::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:49:30,128::__init__::232::IOProcess::(_processLogs) (109537) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:30,136::__init__::232::IOProcess::(_processLogs) (109537) Queuing response >Thread-79::DEBUG::2014-09-25 17:49:30,136::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000413996 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:30,137::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:30,138::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:30,138::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:30,138::__init__::232::IOProcess::(_processLogs) (109538) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:30,139::__init__::232::IOProcess::(_processLogs) (109538) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:30,140::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:30,140::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:30,141::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:30,141::__init__::232::IOProcess::(_processLogs) (109539) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:30,141::__init__::232::IOProcess::(_processLogs) (109539) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:30,142::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:30,142::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:30,143::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:30,143::__init__::232::IOProcess::(_processLogs) (109540) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:30,143::__init__::232::IOProcess::(_processLogs) (109540) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:30,144::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:30,144::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:30,145::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:30,145::__init__::232::IOProcess::(_processLogs) (109541) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:30,145::__init__::232::IOProcess::(_processLogs) (109541) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:30,391::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:30,392::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71196::DEBUG::2014-09-25 17:49:30,392::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71196::DEBUG::2014-09-25 17:49:30,394::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b26e08b-97e2-4dcb-b048-4d4b7cc3fde6`::moving from state init -> state preparing >Thread-71196::INFO::2014-09-25 17:49:30,394::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71196::INFO::2014-09-25 17:49:30,395::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71196::DEBUG::2014-09-25 17:49:30,395::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b26e08b-97e2-4dcb-b048-4d4b7cc3fde6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71196::DEBUG::2014-09-25 17:49:30,395::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b26e08b-97e2-4dcb-b048-4d4b7cc3fde6`::moving from state preparing -> state finished >Thread-71196::DEBUG::2014-09-25 17:49:30,396::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71196::DEBUG::2014-09-25 17:49:30,396::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71196::DEBUG::2014-09-25 17:49:30,396::task::993::Storage.TaskManager.Task::(_decref) Task=`1b26e08b-97e2-4dcb-b048-4d4b7cc3fde6`::ref 0 aborting False >Thread-71196::DEBUG::2014-09-25 17:49:30,396::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71196::DEBUG::2014-09-25 17:49:30,397::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:30,411::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:30,412::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71197::DEBUG::2014-09-25 17:49:30,413::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71197::DEBUG::2014-09-25 17:49:30,414::task::595::Storage.TaskManager.Task::(_updateState) Task=`826c61b9-8956-496a-9904-e0a335b286ce`::moving from state init -> state preparing >Thread-71197::INFO::2014-09-25 17:49:30,414::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71197::DEBUG::2014-09-25 17:49:30,415::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`51e3f155-8420-4982-a6e2-d74ef64115ba`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71197::DEBUG::2014-09-25 17:49:30,415::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71197::DEBUG::2014-09-25 17:49:30,416::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71197::DEBUG::2014-09-25 17:49:30,416::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`51e3f155-8420-4982-a6e2-d74ef64115ba`::Granted request >Thread-71197::DEBUG::2014-09-25 17:49:30,418::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`826c61b9-8956-496a-9904-e0a335b286ce`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71197::DEBUG::2014-09-25 17:49:30,418::task::993::Storage.TaskManager.Task::(_decref) Task=`826c61b9-8956-496a-9904-e0a335b286ce`::ref 1 aborting False >Thread-71197::INFO::2014-09-25 17:49:30,422::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71197::DEBUG::2014-09-25 17:49:30,423::task::1191::Storage.TaskManager.Task::(prepare) Task=`826c61b9-8956-496a-9904-e0a335b286ce`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71197::DEBUG::2014-09-25 17:49:30,423::task::595::Storage.TaskManager.Task::(_updateState) Task=`826c61b9-8956-496a-9904-e0a335b286ce`::moving from state preparing -> state finished >Thread-71197::DEBUG::2014-09-25 17:49:30,423::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71197::DEBUG::2014-09-25 17:49:30,423::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71197::DEBUG::2014-09-25 17:49:30,424::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71197::DEBUG::2014-09-25 17:49:30,424::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71197::DEBUG::2014-09-25 17:49:30,424::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71197::DEBUG::2014-09-25 17:49:30,425::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71197::DEBUG::2014-09-25 17:49:30,425::task::993::Storage.TaskManager.Task::(_decref) Task=`826c61b9-8956-496a-9904-e0a335b286ce`::ref 0 aborting False >Thread-71197::DEBUG::2014-09-25 17:49:30,425::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71197::DEBUG::2014-09-25 17:49:30,426::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:31,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:31,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134064 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:32,466::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:32,468::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71198::DEBUG::2014-09-25 17:49:32,470::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:33,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:33,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136803 s, 74.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:34,512::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:34,513::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:34,513::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:34,513::__init__::232::IOProcess::(_processLogs) (109542) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:34,514::__init__::232::IOProcess::(_processLogs) (109542) Queuing response >Thread-81::DEBUG::2014-09-25 17:49:34,515::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:49:34,524::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00052331 s, 709 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:34,525::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:34,525::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:34,526::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:34,526::__init__::232::IOProcess::(_processLogs) (109543) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:34,527::__init__::232::IOProcess::(_processLogs) (109543) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:35,487::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:35,489::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71199::DEBUG::2014-09-25 17:49:35,491::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:36,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:36,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205001 s, 50.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:36,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:36,653::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:36,653::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:36,653::__init__::232::IOProcess::(_processLogs) (109544) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:36,654::__init__::232::IOProcess::(_processLogs) (109544) Queuing response >Thread-80::DEBUG::2014-09-25 17:49:36,655::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:49:36,664::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000487793 s, 775 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:36,665::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:36,666::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:36,666::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:36,666::__init__::232::IOProcess::(_processLogs) (109545) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:36,667::__init__::232::IOProcess::(_processLogs) (109545) Queuing response >Dummy-148::DEBUG::2014-09-25 17:49:38,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:38,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133418 s, 76.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:38,508::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:38,509::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71200::DEBUG::2014-09-25 17:49:38,511::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:40,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:40,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136263 s, 75.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:40,147::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:40,147::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:40,147::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:40,148::__init__::232::IOProcess::(_processLogs) (109546) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:40,148::__init__::232::IOProcess::(_processLogs) (109546) Queuing response >Thread-79::DEBUG::2014-09-25 17:49:40,149::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:49:40,158::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000474941 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:40,159::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:40,159::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:40,160::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:40,161::__init__::232::IOProcess::(_processLogs) (109547) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:40,161::__init__::232::IOProcess::(_processLogs) (109547) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:40,162::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:40,162::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:40,163::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:40,163::__init__::232::IOProcess::(_processLogs) (109548) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:40,164::__init__::232::IOProcess::(_processLogs) (109548) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:40,164::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:40,164::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:40,165::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:40,165::__init__::232::IOProcess::(_processLogs) (109549) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:40,166::__init__::232::IOProcess::(_processLogs) (109549) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:40,166::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:40,167::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:40,167::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:40,168::__init__::232::IOProcess::(_processLogs) (109550) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:40,168::__init__::232::IOProcess::(_processLogs) (109550) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:40,456::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71201::DEBUG::2014-09-25 17:49:40,457::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:49:40,457::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71201::DEBUG::2014-09-25 17:49:40,460::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f5b2378-463f-48e6-b1ad-33bd8e7e3916`::moving from state init -> state preparing >Thread-71201::INFO::2014-09-25 17:49:40,460::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71201::INFO::2014-09-25 17:49:40,461::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71201::DEBUG::2014-09-25 17:49:40,461::task::1191::Storage.TaskManager.Task::(prepare) Task=`4f5b2378-463f-48e6-b1ad-33bd8e7e3916`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71201::DEBUG::2014-09-25 17:49:40,461::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f5b2378-463f-48e6-b1ad-33bd8e7e3916`::moving from state preparing -> state finished >Thread-71201::DEBUG::2014-09-25 17:49:40,461::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71201::DEBUG::2014-09-25 17:49:40,462::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71201::DEBUG::2014-09-25 17:49:40,462::task::993::Storage.TaskManager.Task::(_decref) Task=`4f5b2378-463f-48e6-b1ad-33bd8e7e3916`::ref 0 aborting False >Thread-71201::DEBUG::2014-09-25 17:49:40,462::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71201::DEBUG::2014-09-25 17:49:40,462::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:40,476::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71202::DEBUG::2014-09-25 17:49:40,478::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:49:40,478::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71202::DEBUG::2014-09-25 17:49:40,480::task::595::Storage.TaskManager.Task::(_updateState) Task=`1485df86-e328-4f86-9b82-43bc3bb7edf4`::moving from state init -> state preparing >Thread-71202::INFO::2014-09-25 17:49:40,480::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71202::DEBUG::2014-09-25 17:49:40,481::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1edd2c68-4f1b-4eeb-974d-6750d598f918`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71202::DEBUG::2014-09-25 17:49:40,481::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71202::DEBUG::2014-09-25 17:49:40,481::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71202::DEBUG::2014-09-25 17:49:40,482::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1edd2c68-4f1b-4eeb-974d-6750d598f918`::Granted request >Thread-71202::DEBUG::2014-09-25 17:49:40,482::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1485df86-e328-4f86-9b82-43bc3bb7edf4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71202::DEBUG::2014-09-25 17:49:40,483::task::993::Storage.TaskManager.Task::(_decref) Task=`1485df86-e328-4f86-9b82-43bc3bb7edf4`::ref 1 aborting False >Thread-71202::INFO::2014-09-25 17:49:40,487::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71202::DEBUG::2014-09-25 17:49:40,487::task::1191::Storage.TaskManager.Task::(prepare) Task=`1485df86-e328-4f86-9b82-43bc3bb7edf4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71202::DEBUG::2014-09-25 17:49:40,487::task::595::Storage.TaskManager.Task::(_updateState) Task=`1485df86-e328-4f86-9b82-43bc3bb7edf4`::moving from state preparing -> state finished >Thread-71202::DEBUG::2014-09-25 17:49:40,488::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71202::DEBUG::2014-09-25 17:49:40,488::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71202::DEBUG::2014-09-25 17:49:40,488::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71202::DEBUG::2014-09-25 17:49:40,489::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71202::DEBUG::2014-09-25 17:49:40,489::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71202::DEBUG::2014-09-25 17:49:40,489::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71202::DEBUG::2014-09-25 17:49:40,490::task::993::Storage.TaskManager.Task::(_decref) Task=`1485df86-e328-4f86-9b82-43bc3bb7edf4`::ref 0 aborting False >Thread-71202::DEBUG::2014-09-25 17:49:40,490::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71202::DEBUG::2014-09-25 17:49:40,491::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:41,526::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:41,527::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71203::DEBUG::2014-09-25 17:49:41,530::task::595::Storage.TaskManager.Task::(_updateState) Task=`79d7d129-4a08-4826-ac0d-0f6f76690bd9`::moving from state init -> state preparing >Thread-71203::INFO::2014-09-25 17:49:41,530::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71203::INFO::2014-09-25 17:49:41,531::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000474941', 'lastCheck': '1.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000487793', 'lastCheck': '4.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00052331', 'lastCheck': '7.0', 'valid': True}} >Thread-71203::DEBUG::2014-09-25 17:49:41,531::task::1191::Storage.TaskManager.Task::(prepare) Task=`79d7d129-4a08-4826-ac0d-0f6f76690bd9`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000474941', 'lastCheck': '1.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000487793', 'lastCheck': '4.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00052331', 'lastCheck': '7.0', 'valid': True}} >Thread-71203::DEBUG::2014-09-25 17:49:41,531::task::595::Storage.TaskManager.Task::(_updateState) Task=`79d7d129-4a08-4826-ac0d-0f6f76690bd9`::moving from state preparing -> state finished >Thread-71203::DEBUG::2014-09-25 17:49:41,532::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71203::DEBUG::2014-09-25 17:49:41,532::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71203::DEBUG::2014-09-25 17:49:41,532::task::993::Storage.TaskManager.Task::(_decref) Task=`79d7d129-4a08-4826-ac0d-0f6f76690bd9`::ref 0 aborting False >Thread-71203::DEBUG::2014-09-25 17:49:41,535::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:41,556::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:41,557::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71204::DEBUG::2014-09-25 17:49:41,559::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:42,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:42,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150525 s, 68.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:49:44,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:44,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137362 s, 74.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:44,529::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:44,529::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:44,529::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:44,530::__init__::232::IOProcess::(_processLogs) (109551) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:44,530::__init__::232::IOProcess::(_processLogs) (109551) Queuing response >Thread-81::DEBUG::2014-09-25 17:49:44,531::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:49:44,540::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000420038 s, 883 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:44,541::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:44,542::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:44,542::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:44,542::__init__::232::IOProcess::(_processLogs) (109552) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:44,543::__init__::232::IOProcess::(_processLogs) (109552) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:44,639::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:44,641::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71205::DEBUG::2014-09-25 17:49:44,643::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:46,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:46,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0277461 s, 36.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:46,670::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:46,671::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:46,673::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:49:46,675::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:49:46,675::__init__::232::IOProcess::(_processLogs) (109553) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:46,685::__init__::232::IOProcess::(_processLogs) (109553) Queuing response >Thread-80::DEBUG::2014-09-25 17:49:46,687::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000384657 s, 983 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:46,690::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:46,691::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:46,692::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:46,693::__init__::232::IOProcess::(_processLogs) (109554) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:46,695::__init__::232::IOProcess::(_processLogs) (109554) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:47,658::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:47,660::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71206::DEBUG::2014-09-25 17:49:47,662::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:48,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:48,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136424 s, 75.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:50,168::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:50,169::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:50,169::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:50,169::__init__::232::IOProcess::(_processLogs) (109555) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:50,170::__init__::232::IOProcess::(_processLogs) (109555) Queuing response >Thread-79::DEBUG::2014-09-25 17:49:50,171::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:49:50,180::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000508497 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:50,181::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:50,182::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:50,182::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:50,182::__init__::232::IOProcess::(_processLogs) (109556) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:50,183::__init__::232::IOProcess::(_processLogs) (109556) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:50,184::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:50,185::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:50,185::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:50,185::__init__::232::IOProcess::(_processLogs) (109557) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:50,186::__init__::232::IOProcess::(_processLogs) (109557) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:50,186::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:50,187::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:50,187::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:50,188::__init__::232::IOProcess::(_processLogs) (109558) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:50,188::__init__::232::IOProcess::(_processLogs) (109558) Queuing response >Thread-29::DEBUG::2014-09-25 17:49:50,189::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:50,189::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:50,189::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:50,190::__init__::232::IOProcess::(_processLogs) (109559) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:49:50,191::__init__::232::IOProcess::(_processLogs) (109559) Queuing response >Dummy-148::DEBUG::2014-09-25 17:49:50,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:50,537::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:50,542::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71207::DEBUG::2014-09-25 17:49:50,542::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71207::DEBUG::2014-09-25 17:49:50,547::task::595::Storage.TaskManager.Task::(_updateState) Task=`08c05888-47f1-43a8-bb92-bb0dd7c64d4b`::moving from state init -> state preparing >Thread-71207::INFO::2014-09-25 17:49:50,548::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Dummy-148::DEBUG::2014-09-25 17:49:50,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0226174 s, 45.3 MB/s\n'; <rc> = 0 >Thread-71207::INFO::2014-09-25 17:49:50,550::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71207::DEBUG::2014-09-25 17:49:50,551::task::1191::Storage.TaskManager.Task::(prepare) Task=`08c05888-47f1-43a8-bb92-bb0dd7c64d4b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71207::DEBUG::2014-09-25 17:49:50,552::task::595::Storage.TaskManager.Task::(_updateState) Task=`08c05888-47f1-43a8-bb92-bb0dd7c64d4b`::moving from state preparing -> state finished >Thread-71207::DEBUG::2014-09-25 17:49:50,553::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71207::DEBUG::2014-09-25 17:49:50,553::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71207::DEBUG::2014-09-25 17:49:50,554::task::993::Storage.TaskManager.Task::(_decref) Task=`08c05888-47f1-43a8-bb92-bb0dd7c64d4b`::ref 0 aborting False >Thread-71207::DEBUG::2014-09-25 17:49:50,556::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71207::DEBUG::2014-09-25 17:49:50,557::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:50,573::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:50,577::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71208::DEBUG::2014-09-25 17:49:50,577::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71208::DEBUG::2014-09-25 17:49:50,581::task::595::Storage.TaskManager.Task::(_updateState) Task=`898315ac-6b8a-4d3c-a927-9fc656634743`::moving from state init -> state preparing >Thread-71208::INFO::2014-09-25 17:49:50,582::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71208::DEBUG::2014-09-25 17:49:50,584::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d02de206-a539-492e-a7af-2babc9c2a92e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71208::DEBUG::2014-09-25 17:49:50,585::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71208::DEBUG::2014-09-25 17:49:50,586::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71208::DEBUG::2014-09-25 17:49:50,587::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`d02de206-a539-492e-a7af-2babc9c2a92e`::Granted request >Thread-71208::DEBUG::2014-09-25 17:49:50,589::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`898315ac-6b8a-4d3c-a927-9fc656634743`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71208::DEBUG::2014-09-25 17:49:50,590::task::993::Storage.TaskManager.Task::(_decref) Task=`898315ac-6b8a-4d3c-a927-9fc656634743`::ref 1 aborting False >Thread-71208::INFO::2014-09-25 17:49:50,606::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71208::DEBUG::2014-09-25 17:49:50,607::task::1191::Storage.TaskManager.Task::(prepare) Task=`898315ac-6b8a-4d3c-a927-9fc656634743`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71208::DEBUG::2014-09-25 17:49:50,607::task::595::Storage.TaskManager.Task::(_updateState) Task=`898315ac-6b8a-4d3c-a927-9fc656634743`::moving from state preparing -> state finished >Thread-71208::DEBUG::2014-09-25 17:49:50,608::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71208::DEBUG::2014-09-25 17:49:50,609::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71208::DEBUG::2014-09-25 17:49:50,611::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71208::DEBUG::2014-09-25 17:49:50,612::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71208::DEBUG::2014-09-25 17:49:50,613::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71208::DEBUG::2014-09-25 17:49:50,614::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71208::DEBUG::2014-09-25 17:49:50,615::task::993::Storage.TaskManager.Task::(_decref) Task=`898315ac-6b8a-4d3c-a927-9fc656634743`::ref 0 aborting False >Thread-71208::DEBUG::2014-09-25 17:49:50,617::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71208::DEBUG::2014-09-25 17:49:50,618::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:50,676::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:50,678::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71209::DEBUG::2014-09-25 17:49:50,679::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:52,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:52,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135926 s, 75.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:53,696::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:53,698::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71210::DEBUG::2014-09-25 17:49:53,700::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:49:54,545::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:54,545::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:54,546::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:54,546::__init__::232::IOProcess::(_processLogs) (109560) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:54,546::__init__::232::IOProcess::(_processLogs) (109560) Queuing response >Thread-81::DEBUG::2014-09-25 17:49:54,548::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:49:54,557::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000507404 s, 731 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:54,558::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:54,558::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:54,559::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:54,560::__init__::232::IOProcess::(_processLogs) (109561) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:54,561::__init__::232::IOProcess::(_processLogs) (109561) Queuing response >Dummy-148::DEBUG::2014-09-25 17:49:54,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:54,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138933 s, 73.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:49:56,699::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:56,699::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:56,699::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:56,700::__init__::232::IOProcess::(_processLogs) (109562) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:56,700::__init__::232::IOProcess::(_processLogs) (109562) Queuing response >Thread-80::DEBUG::2014-09-25 17:49:56,701::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:49:56,710::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000530743 s, 712 kB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:49:56,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:49:56,713::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:49:56,721::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:49:56,721::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:49:56,721::__init__::232::IOProcess::(_processLogs) (109563) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:49:56,722::__init__::232::IOProcess::(_processLogs) (109563) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:56,725::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:56,727::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71211::DEBUG::2014-09-25 17:49:56,731::task::595::Storage.TaskManager.Task::(_updateState) Task=`91adbdd0-b315-4587-8659-bd95933e6de1`::moving from state init -> state preparing >Thread-71211::INFO::2014-09-25 17:49:56,731::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71211::INFO::2014-09-25 17:49:56,733::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000508497', 'lastCheck': '6.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000530743', 'lastCheck': '0.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000507404', 'lastCheck': '2.2', 'valid': True}} >Thread-71211::DEBUG::2014-09-25 17:49:56,734::task::1191::Storage.TaskManager.Task::(prepare) Task=`91adbdd0-b315-4587-8659-bd95933e6de1`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000508497', 'lastCheck': '6.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000530743', 'lastCheck': '0.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000507404', 'lastCheck': '2.2', 'valid': True}} >Thread-71211::DEBUG::2014-09-25 17:49:56,735::task::595::Storage.TaskManager.Task::(_updateState) Task=`91adbdd0-b315-4587-8659-bd95933e6de1`::moving from state preparing -> state finished >Thread-71211::DEBUG::2014-09-25 17:49:56,736::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71211::DEBUG::2014-09-25 17:49:56,737::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71211::DEBUG::2014-09-25 17:49:56,738::task::993::Storage.TaskManager.Task::(_decref) Task=`91adbdd0-b315-4587-8659-bd95933e6de1`::ref 0 aborting False >Thread-71211::DEBUG::2014-09-25 17:49:56,748::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:56,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0267035 s, 38.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:56,772::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:56,773::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71212::DEBUG::2014-09-25 17:49:56,775::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:49:58,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:49:58,803::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166051 s, 61.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:49:59,836::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:49:59,838::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71213::DEBUG::2014-09-25 17:49:59,840::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:50:00,191::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:00,191::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:00,192::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:00,192::__init__::232::IOProcess::(_processLogs) (109564) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:00,192::__init__::232::IOProcess::(_processLogs) (109564) Queuing response >Thread-79::DEBUG::2014-09-25 17:50:00,194::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:50:00,203::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000486198 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:00,204::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:00,204::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:00,204::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:00,205::__init__::232::IOProcess::(_processLogs) (109565) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:00,205::__init__::232::IOProcess::(_processLogs) (109565) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:00,206::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:00,207::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:00,208::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:00,208::__init__::232::IOProcess::(_processLogs) (109566) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:00,208::__init__::232::IOProcess::(_processLogs) (109566) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:00,208::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:00,209::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:00,209::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:00,210::__init__::232::IOProcess::(_processLogs) (109567) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:00,210::__init__::232::IOProcess::(_processLogs) (109567) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:00,211::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:00,211::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:00,212::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:00,212::__init__::232::IOProcess::(_processLogs) (109568) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:00,212::__init__::232::IOProcess::(_processLogs) (109568) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:00,649::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71214::DEBUG::2014-09-25 17:50:00,651::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:50:00,651::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71214::DEBUG::2014-09-25 17:50:00,653::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5c9cbfb-9d25-4c5c-9097-ba92596bfa1c`::moving from state init -> state preparing >Thread-71214::INFO::2014-09-25 17:50:00,654::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71214::INFO::2014-09-25 17:50:00,654::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71214::DEBUG::2014-09-25 17:50:00,655::task::1191::Storage.TaskManager.Task::(prepare) Task=`a5c9cbfb-9d25-4c5c-9097-ba92596bfa1c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71214::DEBUG::2014-09-25 17:50:00,655::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5c9cbfb-9d25-4c5c-9097-ba92596bfa1c`::moving from state preparing -> state finished >Thread-71214::DEBUG::2014-09-25 17:50:00,655::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71214::DEBUG::2014-09-25 17:50:00,655::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71214::DEBUG::2014-09-25 17:50:00,655::task::993::Storage.TaskManager.Task::(_decref) Task=`a5c9cbfb-9d25-4c5c-9097-ba92596bfa1c`::ref 0 aborting False >Thread-71214::DEBUG::2014-09-25 17:50:00,656::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71214::DEBUG::2014-09-25 17:50:00,656::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:00,669::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:00,670::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71215::DEBUG::2014-09-25 17:50:00,671::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71215::DEBUG::2014-09-25 17:50:00,672::task::595::Storage.TaskManager.Task::(_updateState) Task=`a4480c7b-b638-4e38-bffd-73d4dbcba857`::moving from state init -> state preparing >Thread-71215::INFO::2014-09-25 17:50:00,672::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71215::DEBUG::2014-09-25 17:50:00,673::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`42e0fd2d-0e44-4327-a66d-ba8ea938b4f2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71215::DEBUG::2014-09-25 17:50:00,673::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71215::DEBUG::2014-09-25 17:50:00,674::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71215::DEBUG::2014-09-25 17:50:00,674::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`42e0fd2d-0e44-4327-a66d-ba8ea938b4f2`::Granted request >Thread-71215::DEBUG::2014-09-25 17:50:00,675::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a4480c7b-b638-4e38-bffd-73d4dbcba857`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71215::DEBUG::2014-09-25 17:50:00,675::task::993::Storage.TaskManager.Task::(_decref) Task=`a4480c7b-b638-4e38-bffd-73d4dbcba857`::ref 1 aborting False >Thread-71215::INFO::2014-09-25 17:50:00,679::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71215::DEBUG::2014-09-25 17:50:00,679::task::1191::Storage.TaskManager.Task::(prepare) Task=`a4480c7b-b638-4e38-bffd-73d4dbcba857`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71215::DEBUG::2014-09-25 17:50:00,680::task::595::Storage.TaskManager.Task::(_updateState) Task=`a4480c7b-b638-4e38-bffd-73d4dbcba857`::moving from state preparing -> state finished >Thread-71215::DEBUG::2014-09-25 17:50:00,680::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71215::DEBUG::2014-09-25 17:50:00,680::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71215::DEBUG::2014-09-25 17:50:00,680::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71215::DEBUG::2014-09-25 17:50:00,681::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71215::DEBUG::2014-09-25 17:50:00,681::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71215::DEBUG::2014-09-25 17:50:00,681::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71215::DEBUG::2014-09-25 17:50:00,682::task::993::Storage.TaskManager.Task::(_decref) Task=`a4480c7b-b638-4e38-bffd-73d4dbcba857`::ref 0 aborting False >Thread-71215::DEBUG::2014-09-25 17:50:00,682::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71215::DEBUG::2014-09-25 17:50:00,683::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:00,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:00,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142947 s, 71.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:02,858::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:02,860::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71216::DEBUG::2014-09-25 17:50:02,862::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:02,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:02,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182296 s, 56.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:04,561::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:04,561::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:04,562::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:04,562::__init__::232::IOProcess::(_processLogs) (109569) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:04,562::__init__::232::IOProcess::(_processLogs) (109569) Queuing response >Thread-81::DEBUG::2014-09-25 17:50:04,564::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:50:04,573::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000469717 s, 790 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:04,574::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:04,575::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:04,575::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:04,575::__init__::232::IOProcess::(_processLogs) (109570) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:04,576::__init__::232::IOProcess::(_processLogs) (109570) Queuing response >Dummy-148::DEBUG::2014-09-25 17:50:04,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:04,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132647 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:05,879::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:05,880::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71217::DEBUG::2014-09-25 17:50:05,882::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:50:06,725::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:06,726::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:06,726::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:06,726::__init__::232::IOProcess::(_processLogs) (109571) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:06,727::__init__::232::IOProcess::(_processLogs) (109571) Queuing response >Thread-80::DEBUG::2014-09-25 17:50:06,728::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:50:06,737::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000516154 s, 732 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:06,739::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:06,739::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:06,739::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:06,739::__init__::232::IOProcess::(_processLogs) (109572) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:06,740::__init__::232::IOProcess::(_processLogs) (109572) Queuing response >Dummy-148::DEBUG::2014-09-25 17:50:06,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:06,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129234 s, 79.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:08,899::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:08,900::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71218::DEBUG::2014-09-25 17:50:08,903::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:08,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:09,020::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131717 s, 77.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:10,213::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:10,213::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:10,213::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:10,214::__init__::232::IOProcess::(_processLogs) (109573) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:10,217::__init__::232::IOProcess::(_processLogs) (109573) Queuing response >Thread-79::DEBUG::2014-09-25 17:50:10,219::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:50:10,235::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00449952 s, 125 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:10,237::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:10,237::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:10,237::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:10,237::__init__::232::IOProcess::(_processLogs) (109574) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:10,239::__init__::232::IOProcess::(_processLogs) (109574) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:10,240::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:10,241::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:10,241::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:10,241::__init__::232::IOProcess::(_processLogs) (109575) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:10,241::__init__::232::IOProcess::(_processLogs) (109575) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:10,242::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:10,242::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:10,243::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:10,243::__init__::232::IOProcess::(_processLogs) (109576) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:10,243::__init__::232::IOProcess::(_processLogs) (109576) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:10,244::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:10,244::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:10,245::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:10,245::__init__::232::IOProcess::(_processLogs) (109577) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:10,245::__init__::232::IOProcess::(_processLogs) (109577) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:10,714::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71219::DEBUG::2014-09-25 17:50:10,715::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:50:10,715::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71219::DEBUG::2014-09-25 17:50:10,718::task::595::Storage.TaskManager.Task::(_updateState) Task=`ad7dc62e-e129-4490-89a2-2b0ca701766b`::moving from state init -> state preparing >Thread-71219::INFO::2014-09-25 17:50:10,718::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71219::INFO::2014-09-25 17:50:10,719::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71219::DEBUG::2014-09-25 17:50:10,719::task::1191::Storage.TaskManager.Task::(prepare) Task=`ad7dc62e-e129-4490-89a2-2b0ca701766b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71219::DEBUG::2014-09-25 17:50:10,719::task::595::Storage.TaskManager.Task::(_updateState) Task=`ad7dc62e-e129-4490-89a2-2b0ca701766b`::moving from state preparing -> state finished >Thread-71219::DEBUG::2014-09-25 17:50:10,719::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71219::DEBUG::2014-09-25 17:50:10,720::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71219::DEBUG::2014-09-25 17:50:10,720::task::993::Storage.TaskManager.Task::(_decref) Task=`ad7dc62e-e129-4490-89a2-2b0ca701766b`::ref 0 aborting False >Thread-71219::DEBUG::2014-09-25 17:50:10,720::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71219::DEBUG::2014-09-25 17:50:10,721::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:10,734::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:10,736::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71220::DEBUG::2014-09-25 17:50:10,736::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71220::DEBUG::2014-09-25 17:50:10,738::task::595::Storage.TaskManager.Task::(_updateState) Task=`a6af1e4f-b9e2-4e5f-8d49-2c0195c283e6`::moving from state init -> state preparing >Thread-71220::INFO::2014-09-25 17:50:10,738::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71220::DEBUG::2014-09-25 17:50:10,739::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`76814b67-d65c-4676-9105-b6070d20d685`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71220::DEBUG::2014-09-25 17:50:10,739::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71220::DEBUG::2014-09-25 17:50:10,739::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71220::DEBUG::2014-09-25 17:50:10,739::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`76814b67-d65c-4676-9105-b6070d20d685`::Granted request >Thread-71220::DEBUG::2014-09-25 17:50:10,740::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a6af1e4f-b9e2-4e5f-8d49-2c0195c283e6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71220::DEBUG::2014-09-25 17:50:10,740::task::993::Storage.TaskManager.Task::(_decref) Task=`a6af1e4f-b9e2-4e5f-8d49-2c0195c283e6`::ref 1 aborting False >Thread-71220::INFO::2014-09-25 17:50:10,745::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71220::DEBUG::2014-09-25 17:50:10,745::task::1191::Storage.TaskManager.Task::(prepare) Task=`a6af1e4f-b9e2-4e5f-8d49-2c0195c283e6`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71220::DEBUG::2014-09-25 17:50:10,745::task::595::Storage.TaskManager.Task::(_updateState) Task=`a6af1e4f-b9e2-4e5f-8d49-2c0195c283e6`::moving from state preparing -> state finished >Thread-71220::DEBUG::2014-09-25 17:50:10,745::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71220::DEBUG::2014-09-25 17:50:10,746::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71220::DEBUG::2014-09-25 17:50:10,746::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71220::DEBUG::2014-09-25 17:50:10,746::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71220::DEBUG::2014-09-25 17:50:10,747::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71220::DEBUG::2014-09-25 17:50:10,747::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71220::DEBUG::2014-09-25 17:50:10,747::task::993::Storage.TaskManager.Task::(_decref) Task=`a6af1e4f-b9e2-4e5f-8d49-2c0195c283e6`::ref 0 aborting False >Thread-71220::DEBUG::2014-09-25 17:50:10,748::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71220::DEBUG::2014-09-25 17:50:10,748::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:11,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:11,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172202 s, 59.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:11,920::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:11,922::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71221::DEBUG::2014-09-25 17:50:11,924::task::595::Storage.TaskManager.Task::(_updateState) Task=`b91f2d3a-8c2b-412e-b117-e598b66c0c0d`::moving from state init -> state preparing >Thread-71221::INFO::2014-09-25 17:50:11,924::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71221::INFO::2014-09-25 17:50:11,925::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00449952', 'lastCheck': '1.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000516154', 'lastCheck': '5.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000469717', 'lastCheck': '7.3', 'valid': True}} >Thread-71221::DEBUG::2014-09-25 17:50:11,925::task::1191::Storage.TaskManager.Task::(prepare) Task=`b91f2d3a-8c2b-412e-b117-e598b66c0c0d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00449952', 'lastCheck': '1.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000516154', 'lastCheck': '5.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000469717', 'lastCheck': '7.3', 'valid': True}} >Thread-71221::DEBUG::2014-09-25 17:50:11,925::task::595::Storage.TaskManager.Task::(_updateState) Task=`b91f2d3a-8c2b-412e-b117-e598b66c0c0d`::moving from state preparing -> state finished >Thread-71221::DEBUG::2014-09-25 17:50:11,926::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71221::DEBUG::2014-09-25 17:50:11,926::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71221::DEBUG::2014-09-25 17:50:11,926::task::993::Storage.TaskManager.Task::(_decref) Task=`b91f2d3a-8c2b-412e-b117-e598b66c0c0d`::ref 0 aborting False >Thread-71221::DEBUG::2014-09-25 17:50:11,929::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:11,950::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:11,951::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71222::DEBUG::2014-09-25 17:50:11,953::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:13,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:13,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016759 s, 61.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:14,578::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:14,578::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:14,579::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:14,579::__init__::232::IOProcess::(_processLogs) (109578) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:14,579::__init__::232::IOProcess::(_processLogs) (109578) Queuing response >Thread-81::DEBUG::2014-09-25 17:50:14,580::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:50:14,591::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00118512 s, 313 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:14,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:14,592::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:14,593::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:14,594::__init__::232::IOProcess::(_processLogs) (109579) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:14,594::__init__::232::IOProcess::(_processLogs) (109579) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:15,011::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:15,013::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71223::DEBUG::2014-09-25 17:50:15,015::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:15,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:15,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136058 s, 75.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:16,744::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:16,746::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-80::DEBUG::2014-09-25 17:50:16,747::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:50:16,748::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:16,757::__init__::232::IOProcess::(_processLogs) (109580) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:50:16,759::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000425885 s, 888 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:16,759::__init__::232::IOProcess::(_processLogs) (109580) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:16,763::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:16,764::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:16,765::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:16,766::__init__::232::IOProcess::(_processLogs) (109581) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:16,768::__init__::232::IOProcess::(_processLogs) (109581) Queuing response >Dummy-148::DEBUG::2014-09-25 17:50:17,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:17,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130261 s, 78.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:18,031::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:18,032::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71224::DEBUG::2014-09-25 17:50:18,034::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:19,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:19,237::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137924 s, 74.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:20,247::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:20,247::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:20,247::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:20,247::__init__::232::IOProcess::(_processLogs) (109582) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:20,248::__init__::232::IOProcess::(_processLogs) (109582) Queuing response >Thread-79::DEBUG::2014-09-25 17:50:20,249::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:50:20,258::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000400261 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:20,259::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:20,260::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:20,260::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:20,261::__init__::232::IOProcess::(_processLogs) (109583) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:20,262::__init__::232::IOProcess::(_processLogs) (109583) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:20,262::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:20,262::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:20,263::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:20,264::__init__::232::IOProcess::(_processLogs) (109584) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:20,264::__init__::232::IOProcess::(_processLogs) (109584) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:20,264::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:20,264::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:20,265::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:20,265::__init__::232::IOProcess::(_processLogs) (109585) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:20,265::__init__::232::IOProcess::(_processLogs) (109585) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:20,266::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:20,266::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:20,267::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:20,268::__init__::232::IOProcess::(_processLogs) (109586) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:20,268::__init__::232::IOProcess::(_processLogs) (109586) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:20,782::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:20,784::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71225::DEBUG::2014-09-25 17:50:20,784::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71225::DEBUG::2014-09-25 17:50:20,786::task::595::Storage.TaskManager.Task::(_updateState) Task=`444dc1ea-fb12-488c-93a8-527abf9537cf`::moving from state init -> state preparing >Thread-71225::INFO::2014-09-25 17:50:20,786::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71225::INFO::2014-09-25 17:50:20,787::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71225::DEBUG::2014-09-25 17:50:20,787::task::1191::Storage.TaskManager.Task::(prepare) Task=`444dc1ea-fb12-488c-93a8-527abf9537cf`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71225::DEBUG::2014-09-25 17:50:20,787::task::595::Storage.TaskManager.Task::(_updateState) Task=`444dc1ea-fb12-488c-93a8-527abf9537cf`::moving from state preparing -> state finished >Thread-71225::DEBUG::2014-09-25 17:50:20,788::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71225::DEBUG::2014-09-25 17:50:20,788::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71225::DEBUG::2014-09-25 17:50:20,788::task::993::Storage.TaskManager.Task::(_decref) Task=`444dc1ea-fb12-488c-93a8-527abf9537cf`::ref 0 aborting False >Thread-71225::DEBUG::2014-09-25 17:50:20,788::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71225::DEBUG::2014-09-25 17:50:20,789::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:20,802::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:20,803::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71226::DEBUG::2014-09-25 17:50:20,803::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71226::DEBUG::2014-09-25 17:50:20,805::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b41289d-792b-4b34-bd60-35f2edd8937c`::moving from state init -> state preparing >Thread-71226::INFO::2014-09-25 17:50:20,805::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71226::DEBUG::2014-09-25 17:50:20,806::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aa859d62-901a-4eb6-9b1c-778957fbf260`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71226::DEBUG::2014-09-25 17:50:20,806::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71226::DEBUG::2014-09-25 17:50:20,806::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71226::DEBUG::2014-09-25 17:50:20,807::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`aa859d62-901a-4eb6-9b1c-778957fbf260`::Granted request >Thread-71226::DEBUG::2014-09-25 17:50:20,807::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4b41289d-792b-4b34-bd60-35f2edd8937c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71226::DEBUG::2014-09-25 17:50:20,807::task::993::Storage.TaskManager.Task::(_decref) Task=`4b41289d-792b-4b34-bd60-35f2edd8937c`::ref 1 aborting False >Thread-71226::INFO::2014-09-25 17:50:20,812::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71226::DEBUG::2014-09-25 17:50:20,812::task::1191::Storage.TaskManager.Task::(prepare) Task=`4b41289d-792b-4b34-bd60-35f2edd8937c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71226::DEBUG::2014-09-25 17:50:20,812::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b41289d-792b-4b34-bd60-35f2edd8937c`::moving from state preparing -> state finished >Thread-71226::DEBUG::2014-09-25 17:50:20,813::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71226::DEBUG::2014-09-25 17:50:20,813::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71226::DEBUG::2014-09-25 17:50:20,813::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71226::DEBUG::2014-09-25 17:50:20,813::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71226::DEBUG::2014-09-25 17:50:20,814::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71226::DEBUG::2014-09-25 17:50:20,814::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71226::DEBUG::2014-09-25 17:50:20,814::task::993::Storage.TaskManager.Task::(_decref) Task=`4b41289d-792b-4b34-bd60-35f2edd8937c`::ref 0 aborting False >Thread-71226::DEBUG::2014-09-25 17:50:20,815::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71226::DEBUG::2014-09-25 17:50:20,815::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:21,049::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:21,051::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71227::DEBUG::2014-09-25 17:50:21,053::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:21,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:21,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133561 s, 76.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:50:23,298::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:23,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133663 s, 76.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:24,069::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:24,070::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71228::DEBUG::2014-09-25 17:50:24,072::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:50:24,595::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:24,595::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:24,595::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:24,596::__init__::232::IOProcess::(_processLogs) (109587) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:24,596::__init__::232::IOProcess::(_processLogs) (109587) Queuing response >Thread-81::DEBUG::2014-09-25 17:50:24,598::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:50:24,607::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000313005 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:24,608::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:24,608::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:24,608::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:24,609::__init__::232::IOProcess::(_processLogs) (109588) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:24,610::__init__::232::IOProcess::(_processLogs) (109588) Queuing response >Dummy-148::DEBUG::2014-09-25 17:50:25,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:25,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134194 s, 76.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:26,770::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:26,771::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:26,771::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:26,771::__init__::232::IOProcess::(_processLogs) (109589) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:26,772::__init__::232::IOProcess::(_processLogs) (109589) Queuing response >Thread-80::DEBUG::2014-09-25 17:50:26,773::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:50:26,782::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000453682 s, 833 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:26,783::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:26,783::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:26,784::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:26,784::__init__::232::IOProcess::(_processLogs) (109590) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:26,785::__init__::232::IOProcess::(_processLogs) (109590) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:27,087::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:27,088::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71229::DEBUG::2014-09-25 17:50:27,091::task::595::Storage.TaskManager.Task::(_updateState) Task=`301313ff-3ffc-42ab-a6e9-8a38c59d32c6`::moving from state init -> state preparing >Thread-71229::INFO::2014-09-25 17:50:27,092::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71229::INFO::2014-09-25 17:50:27,092::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000400261', 'lastCheck': '6.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000453682', 'lastCheck': '0.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000313005', 'lastCheck': '2.5', 'valid': True}} >Thread-71229::DEBUG::2014-09-25 17:50:27,092::task::1191::Storage.TaskManager.Task::(prepare) Task=`301313ff-3ffc-42ab-a6e9-8a38c59d32c6`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000400261', 'lastCheck': '6.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000453682', 'lastCheck': '0.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000313005', 'lastCheck': '2.5', 'valid': True}} >Thread-71229::DEBUG::2014-09-25 17:50:27,093::task::595::Storage.TaskManager.Task::(_updateState) Task=`301313ff-3ffc-42ab-a6e9-8a38c59d32c6`::moving from state preparing -> state finished >Thread-71229::DEBUG::2014-09-25 17:50:27,093::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71229::DEBUG::2014-09-25 17:50:27,093::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71229::DEBUG::2014-09-25 17:50:27,093::task::993::Storage.TaskManager.Task::(_decref) Task=`301313ff-3ffc-42ab-a6e9-8a38c59d32c6`::ref 0 aborting False >Thread-71229::DEBUG::2014-09-25 17:50:27,096::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:27,114::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:27,115::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71230::DEBUG::2014-09-25 17:50:27,117::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:27,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:27,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130815 s, 78.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:50:29,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:29,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131925 s, 77.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:30,179::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:30,180::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71231::DEBUG::2014-09-25 17:50:30,183::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:50:30,268::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:30,269::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:30,269::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:30,269::__init__::232::IOProcess::(_processLogs) (109591) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:30,269::__init__::232::IOProcess::(_processLogs) (109591) Queuing response >Thread-79::DEBUG::2014-09-25 17:50:30,271::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:50:30,280::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00049179 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:30,281::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:30,282::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:30,283::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:30,283::__init__::232::IOProcess::(_processLogs) (109592) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:30,283::__init__::232::IOProcess::(_processLogs) (109592) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:30,284::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:30,284::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:30,285::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:30,285::__init__::232::IOProcess::(_processLogs) (109593) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:30,285::__init__::232::IOProcess::(_processLogs) (109593) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:30,286::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:30,286::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:30,287::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:30,287::__init__::232::IOProcess::(_processLogs) (109594) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:30,288::__init__::232::IOProcess::(_processLogs) (109594) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:30,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:30,288::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:30,289::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:30,290::__init__::232::IOProcess::(_processLogs) (109595) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:30,290::__init__::232::IOProcess::(_processLogs) (109595) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:30,850::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71232::DEBUG::2014-09-25 17:50:30,851::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:50:30,851::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71232::DEBUG::2014-09-25 17:50:30,853::task::595::Storage.TaskManager.Task::(_updateState) Task=`4bc21ca0-b4ad-4338-b43c-1ed9762f01e5`::moving from state init -> state preparing >Thread-71232::INFO::2014-09-25 17:50:30,854::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71232::INFO::2014-09-25 17:50:30,854::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71232::DEBUG::2014-09-25 17:50:30,854::task::1191::Storage.TaskManager.Task::(prepare) Task=`4bc21ca0-b4ad-4338-b43c-1ed9762f01e5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71232::DEBUG::2014-09-25 17:50:30,855::task::595::Storage.TaskManager.Task::(_updateState) Task=`4bc21ca0-b4ad-4338-b43c-1ed9762f01e5`::moving from state preparing -> state finished >Thread-71232::DEBUG::2014-09-25 17:50:30,855::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71232::DEBUG::2014-09-25 17:50:30,855::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71232::DEBUG::2014-09-25 17:50:30,855::task::993::Storage.TaskManager.Task::(_decref) Task=`4bc21ca0-b4ad-4338-b43c-1ed9762f01e5`::ref 0 aborting False >Thread-71232::DEBUG::2014-09-25 17:50:30,855::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71232::DEBUG::2014-09-25 17:50:30,856::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:30,869::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71233::DEBUG::2014-09-25 17:50:30,870::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:50:30,870::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71233::DEBUG::2014-09-25 17:50:30,872::task::595::Storage.TaskManager.Task::(_updateState) Task=`81fb6430-fe8d-4616-9b3b-b20a752674ef`::moving from state init -> state preparing >Thread-71233::INFO::2014-09-25 17:50:30,873::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71233::DEBUG::2014-09-25 17:50:30,873::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`59d93cb1-34f7-4298-97d9-2539a1cdab93`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71233::DEBUG::2014-09-25 17:50:30,874::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71233::DEBUG::2014-09-25 17:50:30,874::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71233::DEBUG::2014-09-25 17:50:30,874::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`59d93cb1-34f7-4298-97d9-2539a1cdab93`::Granted request >Thread-71233::DEBUG::2014-09-25 17:50:30,875::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`81fb6430-fe8d-4616-9b3b-b20a752674ef`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71233::DEBUG::2014-09-25 17:50:30,875::task::993::Storage.TaskManager.Task::(_decref) Task=`81fb6430-fe8d-4616-9b3b-b20a752674ef`::ref 1 aborting False >Thread-71233::INFO::2014-09-25 17:50:30,879::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7248805888', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71233::DEBUG::2014-09-25 17:50:30,880::task::1191::Storage.TaskManager.Task::(prepare) Task=`81fb6430-fe8d-4616-9b3b-b20a752674ef`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7248805888', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71233::DEBUG::2014-09-25 17:50:30,880::task::595::Storage.TaskManager.Task::(_updateState) Task=`81fb6430-fe8d-4616-9b3b-b20a752674ef`::moving from state preparing -> state finished >Thread-71233::DEBUG::2014-09-25 17:50:30,880::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71233::DEBUG::2014-09-25 17:50:30,880::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71233::DEBUG::2014-09-25 17:50:30,881::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71233::DEBUG::2014-09-25 17:50:30,881::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71233::DEBUG::2014-09-25 17:50:30,881::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71233::DEBUG::2014-09-25 17:50:30,882::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71233::DEBUG::2014-09-25 17:50:30,882::task::993::Storage.TaskManager.Task::(_decref) Task=`81fb6430-fe8d-4616-9b3b-b20a752674ef`::ref 0 aborting False >Thread-71233::DEBUG::2014-09-25 17:50:30,882::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7248805888', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71233::DEBUG::2014-09-25 17:50:30,883::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:31,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:31,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135487 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:33,197::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:33,199::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71234::DEBUG::2014-09-25 17:50:33,201::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:33,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:33,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135055 s, 75.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:34,611::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:34,612::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:34,612::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:34,612::__init__::232::IOProcess::(_processLogs) (109596) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:34,613::__init__::232::IOProcess::(_processLogs) (109596) Queuing response >Thread-81::DEBUG::2014-09-25 17:50:34,614::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:50:34,622::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000261627 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:34,625::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:34,626::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:34,626::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:34,626::__init__::232::IOProcess::(_processLogs) (109597) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:34,626::__init__::232::IOProcess::(_processLogs) (109597) Queuing response >Dummy-148::DEBUG::2014-09-25 17:50:35,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:35,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0216658 s, 47.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:36,216::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:36,217::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71235::DEBUG::2014-09-25 17:50:36,219::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:50:36,787::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:36,787::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:36,787::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:36,788::__init__::232::IOProcess::(_processLogs) (109598) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:50:36,789::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:50:36,789::__init__::232::IOProcess::(_processLogs) (109598) Queuing response >Thread-80::DEBUG::2014-09-25 17:50:36,799::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000453213 s, 834 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:36,800::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:36,800::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:36,800::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:36,800::__init__::232::IOProcess::(_processLogs) (109599) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:36,801::__init__::232::IOProcess::(_processLogs) (109599) Queuing response >Dummy-148::DEBUG::2014-09-25 17:50:37,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:37,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0223116 s, 45.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:39,235::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:39,236::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71236::DEBUG::2014-09-25 17:50:39,238::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:39,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:39,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142911 s, 71.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:40,291::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:40,291::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:40,291::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:40,291::__init__::232::IOProcess::(_processLogs) (109600) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:40,292::__init__::232::IOProcess::(_processLogs) (109600) Queuing response >Thread-79::DEBUG::2014-09-25 17:50:40,293::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:50:40,302::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000525515 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:40,303::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:40,303::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:40,304::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:40,304::__init__::232::IOProcess::(_processLogs) (109601) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:40,305::__init__::232::IOProcess::(_processLogs) (109601) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:40,306::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:40,306::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:40,307::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:40,307::__init__::232::IOProcess::(_processLogs) (109602) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:40,307::__init__::232::IOProcess::(_processLogs) (109602) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:40,308::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:40,308::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:40,309::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:40,309::__init__::232::IOProcess::(_processLogs) (109603) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:40,310::__init__::232::IOProcess::(_processLogs) (109603) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:40,310::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:40,310::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:40,311::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:40,312::__init__::232::IOProcess::(_processLogs) (109604) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:40,312::__init__::232::IOProcess::(_processLogs) (109604) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:40,911::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:40,913::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71237::DEBUG::2014-09-25 17:50:40,913::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71237::DEBUG::2014-09-25 17:50:40,915::task::595::Storage.TaskManager.Task::(_updateState) Task=`0dfea5dc-a72d-4904-ae6b-82548093fd78`::moving from state init -> state preparing >Thread-71237::INFO::2014-09-25 17:50:40,915::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71237::INFO::2014-09-25 17:50:40,916::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71237::DEBUG::2014-09-25 17:50:40,916::task::1191::Storage.TaskManager.Task::(prepare) Task=`0dfea5dc-a72d-4904-ae6b-82548093fd78`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71237::DEBUG::2014-09-25 17:50:40,916::task::595::Storage.TaskManager.Task::(_updateState) Task=`0dfea5dc-a72d-4904-ae6b-82548093fd78`::moving from state preparing -> state finished >Thread-71237::DEBUG::2014-09-25 17:50:40,916::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71237::DEBUG::2014-09-25 17:50:40,917::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71237::DEBUG::2014-09-25 17:50:40,917::task::993::Storage.TaskManager.Task::(_decref) Task=`0dfea5dc-a72d-4904-ae6b-82548093fd78`::ref 0 aborting False >Thread-71237::DEBUG::2014-09-25 17:50:40,917::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71237::DEBUG::2014-09-25 17:50:40,918::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:40,932::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:40,934::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71238::DEBUG::2014-09-25 17:50:40,934::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71238::DEBUG::2014-09-25 17:50:40,935::task::595::Storage.TaskManager.Task::(_updateState) Task=`1cda8446-1bb4-4b8d-9c97-9c81868c7069`::moving from state init -> state preparing >Thread-71238::INFO::2014-09-25 17:50:40,936::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71238::DEBUG::2014-09-25 17:50:40,936::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6bc1abd0-d41c-44e8-94e9-c6a100444d4f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71238::DEBUG::2014-09-25 17:50:40,937::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71238::DEBUG::2014-09-25 17:50:40,937::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71238::DEBUG::2014-09-25 17:50:40,937::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6bc1abd0-d41c-44e8-94e9-c6a100444d4f`::Granted request >Thread-71238::DEBUG::2014-09-25 17:50:40,938::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1cda8446-1bb4-4b8d-9c97-9c81868c7069`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71238::DEBUG::2014-09-25 17:50:40,938::task::993::Storage.TaskManager.Task::(_decref) Task=`1cda8446-1bb4-4b8d-9c97-9c81868c7069`::ref 1 aborting False >Thread-71238::INFO::2014-09-25 17:50:40,942::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71238::DEBUG::2014-09-25 17:50:40,943::task::1191::Storage.TaskManager.Task::(prepare) Task=`1cda8446-1bb4-4b8d-9c97-9c81868c7069`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71238::DEBUG::2014-09-25 17:50:40,943::task::595::Storage.TaskManager.Task::(_updateState) Task=`1cda8446-1bb4-4b8d-9c97-9c81868c7069`::moving from state preparing -> state finished >Thread-71238::DEBUG::2014-09-25 17:50:40,943::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71238::DEBUG::2014-09-25 17:50:40,943::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71238::DEBUG::2014-09-25 17:50:40,944::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71238::DEBUG::2014-09-25 17:50:40,944::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71238::DEBUG::2014-09-25 17:50:40,944::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71238::DEBUG::2014-09-25 17:50:40,945::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71238::DEBUG::2014-09-25 17:50:40,945::task::993::Storage.TaskManager.Task::(_decref) Task=`1cda8446-1bb4-4b8d-9c97-9c81868c7069`::ref 0 aborting False >Thread-71238::DEBUG::2014-09-25 17:50:40,945::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71238::DEBUG::2014-09-25 17:50:40,946::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:41,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:41,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136166 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:42,256::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:42,258::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71239::DEBUG::2014-09-25 17:50:42,260::task::595::Storage.TaskManager.Task::(_updateState) Task=`8030fa7a-d141-4ca4-a8ef-273a122284f0`::moving from state init -> state preparing >Thread-71239::INFO::2014-09-25 17:50:42,261::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71239::INFO::2014-09-25 17:50:42,261::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000525515', 'lastCheck': '1.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000453213', 'lastCheck': '5.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000261627', 'lastCheck': '7.6', 'valid': True}} >Thread-71239::DEBUG::2014-09-25 17:50:42,261::task::1191::Storage.TaskManager.Task::(prepare) Task=`8030fa7a-d141-4ca4-a8ef-273a122284f0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000525515', 'lastCheck': '1.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000453213', 'lastCheck': '5.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000261627', 'lastCheck': '7.6', 'valid': True}} >Thread-71239::DEBUG::2014-09-25 17:50:42,262::task::595::Storage.TaskManager.Task::(_updateState) Task=`8030fa7a-d141-4ca4-a8ef-273a122284f0`::moving from state preparing -> state finished >Thread-71239::DEBUG::2014-09-25 17:50:42,262::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71239::DEBUG::2014-09-25 17:50:42,262::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71239::DEBUG::2014-09-25 17:50:42,262::task::993::Storage.TaskManager.Task::(_decref) Task=`8030fa7a-d141-4ca4-a8ef-273a122284f0`::ref 0 aborting False >Thread-71239::DEBUG::2014-09-25 17:50:42,265::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:42,287::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:42,288::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71240::DEBUG::2014-09-25 17:50:42,290::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:43,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:43,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013365 s, 76.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:44,629::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:44,629::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:44,629::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:44,630::__init__::232::IOProcess::(_processLogs) (109605) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:44,630::__init__::232::IOProcess::(_processLogs) (109605) Queuing response >Thread-81::DEBUG::2014-09-25 17:50:44,631::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:50:44,640::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000343757 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:44,642::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:44,642::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:44,643::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:44,644::__init__::232::IOProcess::(_processLogs) (109606) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:44,644::__init__::232::IOProcess::(_processLogs) (109606) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:45,356::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:45,358::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71241::DEBUG::2014-09-25 17:50:45,360::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:45,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:45,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0140739 s, 72.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:46,805::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:46,806::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:46,807::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:46,808::__init__::232::IOProcess::(_processLogs) (109607) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:46,810::__init__::232::IOProcess::(_processLogs) (109607) Queuing response >Thread-80::DEBUG::2014-09-25 17:50:46,812::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:50:46,824::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000467272 s, 809 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:46,828::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:46,830::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:46,832::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:46,834::__init__::232::IOProcess::(_processLogs) (109608) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:46,835::__init__::232::IOProcess::(_processLogs) (109608) Queuing response >Dummy-148::DEBUG::2014-09-25 17:50:47,832::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:47,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135392 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:48,380::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:48,381::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71242::DEBUG::2014-09-25 17:50:48,383::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:49,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:49,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134349 s, 76.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:50,312::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:50,313::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:50,313::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:50,313::__init__::232::IOProcess::(_processLogs) (109609) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:50,314::__init__::232::IOProcess::(_processLogs) (109609) Queuing response >Thread-79::DEBUG::2014-09-25 17:50:50,315::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:50:50,324::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00050584 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:50,325::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:50,325::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:50,325::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:50,326::__init__::232::IOProcess::(_processLogs) (109610) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:50,326::__init__::232::IOProcess::(_processLogs) (109610) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:50,327::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:50,327::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:50,328::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:50,328::__init__::232::IOProcess::(_processLogs) (109611) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:50,328::__init__::232::IOProcess::(_processLogs) (109611) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:50,329::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:50,329::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:50,330::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:50,330::__init__::232::IOProcess::(_processLogs) (109612) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:50,330::__init__::232::IOProcess::(_processLogs) (109612) Queuing response >Thread-29::DEBUG::2014-09-25 17:50:50,331::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:50,332::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:50,332::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:50,332::__init__::232::IOProcess::(_processLogs) (109613) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:50:50,333::__init__::232::IOProcess::(_processLogs) (109613) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:50,974::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:50,976::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71243::DEBUG::2014-09-25 17:50:50,976::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71243::DEBUG::2014-09-25 17:50:50,978::task::595::Storage.TaskManager.Task::(_updateState) Task=`a013b3d6-ddef-4d0e-90f7-2dca9c4c81c6`::moving from state init -> state preparing >Thread-71243::INFO::2014-09-25 17:50:50,978::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71243::INFO::2014-09-25 17:50:50,979::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71243::DEBUG::2014-09-25 17:50:50,979::task::1191::Storage.TaskManager.Task::(prepare) Task=`a013b3d6-ddef-4d0e-90f7-2dca9c4c81c6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71243::DEBUG::2014-09-25 17:50:50,979::task::595::Storage.TaskManager.Task::(_updateState) Task=`a013b3d6-ddef-4d0e-90f7-2dca9c4c81c6`::moving from state preparing -> state finished >Thread-71243::DEBUG::2014-09-25 17:50:50,979::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71243::DEBUG::2014-09-25 17:50:50,980::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71243::DEBUG::2014-09-25 17:50:50,980::task::993::Storage.TaskManager.Task::(_decref) Task=`a013b3d6-ddef-4d0e-90f7-2dca9c4c81c6`::ref 0 aborting False >Thread-71243::DEBUG::2014-09-25 17:50:50,980::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71243::DEBUG::2014-09-25 17:50:50,981::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:50,995::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:50,996::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71244::DEBUG::2014-09-25 17:50:50,996::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71244::DEBUG::2014-09-25 17:50:50,998::task::595::Storage.TaskManager.Task::(_updateState) Task=`7fa760a1-cad1-4aac-94ff-4560ef09d271`::moving from state init -> state preparing >Thread-71244::INFO::2014-09-25 17:50:50,998::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71244::DEBUG::2014-09-25 17:50:50,999::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c8170dcc-02b4-4595-8bfb-57b00a7d5f5f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71244::DEBUG::2014-09-25 17:50:50,999::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71244::DEBUG::2014-09-25 17:50:51,000::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71244::DEBUG::2014-09-25 17:50:51,000::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`c8170dcc-02b4-4595-8bfb-57b00a7d5f5f`::Granted request >Thread-71244::DEBUG::2014-09-25 17:50:51,001::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7fa760a1-cad1-4aac-94ff-4560ef09d271`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71244::DEBUG::2014-09-25 17:50:51,001::task::993::Storage.TaskManager.Task::(_decref) Task=`7fa760a1-cad1-4aac-94ff-4560ef09d271`::ref 1 aborting False >Thread-71244::INFO::2014-09-25 17:50:51,005::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71244::DEBUG::2014-09-25 17:50:51,005::task::1191::Storage.TaskManager.Task::(prepare) Task=`7fa760a1-cad1-4aac-94ff-4560ef09d271`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71244::DEBUG::2014-09-25 17:50:51,006::task::595::Storage.TaskManager.Task::(_updateState) Task=`7fa760a1-cad1-4aac-94ff-4560ef09d271`::moving from state preparing -> state finished >Thread-71244::DEBUG::2014-09-25 17:50:51,006::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71244::DEBUG::2014-09-25 17:50:51,006::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71244::DEBUG::2014-09-25 17:50:51,006::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71244::DEBUG::2014-09-25 17:50:51,007::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71244::DEBUG::2014-09-25 17:50:51,007::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71244::DEBUG::2014-09-25 17:50:51,007::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71244::DEBUG::2014-09-25 17:50:51,008::task::993::Storage.TaskManager.Task::(_decref) Task=`7fa760a1-cad1-4aac-94ff-4560ef09d271`::ref 0 aborting False >Thread-71244::DEBUG::2014-09-25 17:50:51,008::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71244::DEBUG::2014-09-25 17:50:51,009::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:51,400::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:51,402::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71245::DEBUG::2014-09-25 17:50:51,404::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:51,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:51,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179011 s, 57.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:50:53,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:53,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013458 s, 76.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:54,419::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:54,420::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71246::DEBUG::2014-09-25 17:50:54,423::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:50:54,645::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:54,645::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:54,645::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:54,645::__init__::232::IOProcess::(_processLogs) (109614) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:54,646::__init__::232::IOProcess::(_processLogs) (109614) Queuing response >Thread-81::DEBUG::2014-09-25 17:50:54,647::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:50:54,656::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000487861 s, 760 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:54,657::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:54,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:54,658::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:54,658::__init__::232::IOProcess::(_processLogs) (109615) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:54,659::__init__::232::IOProcess::(_processLogs) (109615) Queuing response >Dummy-148::DEBUG::2014-09-25 17:50:56,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:56,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136132 s, 75.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:56,834::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:56,835::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:56,835::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:56,835::__init__::232::IOProcess::(_processLogs) (109616) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:56,836::__init__::232::IOProcess::(_processLogs) (109616) Queuing response >Thread-80::DEBUG::2014-09-25 17:50:56,837::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:50:56,846::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000428036 s, 883 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:50:56,847::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:50:56,847::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:50:56,847::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:50:56,848::__init__::232::IOProcess::(_processLogs) (109617) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:50:56,848::__init__::232::IOProcess::(_processLogs) (109617) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:57,438::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:57,439::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71247::DEBUG::2014-09-25 17:50:57,442::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f4b452f-6fe3-4d87-8f01-ae1ef432be4d`::moving from state init -> state preparing >Thread-71247::INFO::2014-09-25 17:50:57,442::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71247::INFO::2014-09-25 17:50:57,443::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00050584', 'lastCheck': '7.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000428036', 'lastCheck': '0.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000487861', 'lastCheck': '2.8', 'valid': True}} >Thread-71247::DEBUG::2014-09-25 17:50:57,443::task::1191::Storage.TaskManager.Task::(prepare) Task=`4f4b452f-6fe3-4d87-8f01-ae1ef432be4d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00050584', 'lastCheck': '7.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000428036', 'lastCheck': '0.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000487861', 'lastCheck': '2.8', 'valid': True}} >Thread-71247::DEBUG::2014-09-25 17:50:57,443::task::595::Storage.TaskManager.Task::(_updateState) Task=`4f4b452f-6fe3-4d87-8f01-ae1ef432be4d`::moving from state preparing -> state finished >Thread-71247::DEBUG::2014-09-25 17:50:57,443::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71247::DEBUG::2014-09-25 17:50:57,444::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71247::DEBUG::2014-09-25 17:50:57,444::task::993::Storage.TaskManager.Task::(_decref) Task=`4f4b452f-6fe3-4d87-8f01-ae1ef432be4d`::ref 0 aborting False >Thread-71247::DEBUG::2014-09-25 17:50:57,446::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:50:57,471::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:50:57,472::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71248::DEBUG::2014-09-25 17:50:57,474::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:50:58,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:50:58,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132987 s, 77.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:51:00,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:00,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133 s, 77.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:00,334::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:00,335::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:00,335::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:00,335::__init__::232::IOProcess::(_processLogs) (109618) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:00,335::__init__::232::IOProcess::(_processLogs) (109618) Queuing response >Thread-79::DEBUG::2014-09-25 17:51:00,337::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:51:00,345::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000478502 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:00,347::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:00,347::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:00,348::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:00,349::__init__::232::IOProcess::(_processLogs) (109619) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:00,349::__init__::232::IOProcess::(_processLogs) (109619) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:00,349::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:00,350::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:00,350::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:00,351::__init__::232::IOProcess::(_processLogs) (109620) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:00,351::__init__::232::IOProcess::(_processLogs) (109620) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:00,351::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:00,352::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:00,353::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:00,353::__init__::232::IOProcess::(_processLogs) (109621) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:00,353::__init__::232::IOProcess::(_processLogs) (109621) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:00,354::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:00,354::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:00,354::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:00,355::__init__::232::IOProcess::(_processLogs) (109622) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:00,355::__init__::232::IOProcess::(_processLogs) (109622) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:00,532::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:00,534::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71249::DEBUG::2014-09-25 17:51:00,536::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:01,036::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71250::DEBUG::2014-09-25 17:51:01,037::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:51:01,038::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71250::DEBUG::2014-09-25 17:51:01,040::task::595::Storage.TaskManager.Task::(_updateState) Task=`43fb80e6-126b-439a-9871-628747102b2b`::moving from state init -> state preparing >Thread-71250::INFO::2014-09-25 17:51:01,040::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71250::INFO::2014-09-25 17:51:01,041::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71250::DEBUG::2014-09-25 17:51:01,041::task::1191::Storage.TaskManager.Task::(prepare) Task=`43fb80e6-126b-439a-9871-628747102b2b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71250::DEBUG::2014-09-25 17:51:01,041::task::595::Storage.TaskManager.Task::(_updateState) Task=`43fb80e6-126b-439a-9871-628747102b2b`::moving from state preparing -> state finished >Thread-71250::DEBUG::2014-09-25 17:51:01,041::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71250::DEBUG::2014-09-25 17:51:01,041::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71250::DEBUG::2014-09-25 17:51:01,042::task::993::Storage.TaskManager.Task::(_decref) Task=`43fb80e6-126b-439a-9871-628747102b2b`::ref 0 aborting False >Thread-71250::DEBUG::2014-09-25 17:51:01,042::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71250::DEBUG::2014-09-25 17:51:01,042::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:01,056::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71251::DEBUG::2014-09-25 17:51:01,057::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:51:01,057::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71251::DEBUG::2014-09-25 17:51:01,059::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcd8ca32-9821-4a84-97a3-853cfcef3dec`::moving from state init -> state preparing >Thread-71251::INFO::2014-09-25 17:51:01,059::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71251::DEBUG::2014-09-25 17:51:01,060::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`12f1e0e8-8af8-4612-877b-f1e7267909c5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71251::DEBUG::2014-09-25 17:51:01,060::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71251::DEBUG::2014-09-25 17:51:01,061::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71251::DEBUG::2014-09-25 17:51:01,061::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`12f1e0e8-8af8-4612-877b-f1e7267909c5`::Granted request >Thread-71251::DEBUG::2014-09-25 17:51:01,062::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`fcd8ca32-9821-4a84-97a3-853cfcef3dec`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71251::DEBUG::2014-09-25 17:51:01,062::task::993::Storage.TaskManager.Task::(_decref) Task=`fcd8ca32-9821-4a84-97a3-853cfcef3dec`::ref 1 aborting False >Thread-71251::INFO::2014-09-25 17:51:01,066::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71251::DEBUG::2014-09-25 17:51:01,066::task::1191::Storage.TaskManager.Task::(prepare) Task=`fcd8ca32-9821-4a84-97a3-853cfcef3dec`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71251::DEBUG::2014-09-25 17:51:01,067::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcd8ca32-9821-4a84-97a3-853cfcef3dec`::moving from state preparing -> state finished >Thread-71251::DEBUG::2014-09-25 17:51:01,067::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71251::DEBUG::2014-09-25 17:51:01,067::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71251::DEBUG::2014-09-25 17:51:01,068::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71251::DEBUG::2014-09-25 17:51:01,068::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71251::DEBUG::2014-09-25 17:51:01,068::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71251::DEBUG::2014-09-25 17:51:01,068::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71251::DEBUG::2014-09-25 17:51:01,069::task::993::Storage.TaskManager.Task::(_decref) Task=`fcd8ca32-9821-4a84-97a3-853cfcef3dec`::ref 0 aborting False >Thread-71251::DEBUG::2014-09-25 17:51:01,069::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71251::DEBUG::2014-09-25 17:51:01,070::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:02,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:02,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013616 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:03,553::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:03,555::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71252::DEBUG::2014-09-25 17:51:03,557::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:04,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:04,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019116 s, 53.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:04,661::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:04,661::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:04,662::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:04,662::__init__::232::IOProcess::(_processLogs) (109623) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:04,671::__init__::232::IOProcess::(_processLogs) (109623) Queuing response >Thread-81::DEBUG::2014-09-25 17:51:04,672::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:51:04,694::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00480532 s, 77.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:04,695::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:04,696::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:04,696::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:04,696::__init__::232::IOProcess::(_processLogs) (109624) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:04,699::__init__::232::IOProcess::(_processLogs) (109624) Queuing response >Dummy-148::DEBUG::2014-09-25 17:51:06,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:06,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133082 s, 76.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:06,573::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:06,575::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71253::DEBUG::2014-09-25 17:51:06,577::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:51:06,850::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:06,850::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:06,851::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:06,851::__init__::232::IOProcess::(_processLogs) (109625) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:06,851::__init__::232::IOProcess::(_processLogs) (109625) Queuing response >Thread-80::DEBUG::2014-09-25 17:51:06,853::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:51:06,862::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000560617 s, 674 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:06,863::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:06,863::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:06,863::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:06,864::__init__::232::IOProcess::(_processLogs) (109626) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:06,864::__init__::232::IOProcess::(_processLogs) (109626) Queuing response >Dummy-148::DEBUG::2014-09-25 17:51:08,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:08,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136006 s, 75.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:09,593::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:09,594::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71254::DEBUG::2014-09-25 17:51:09,596::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:10,317::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:10,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135177 s, 75.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:10,357::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:10,358::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:10,359::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:10,360::__init__::232::IOProcess::(_processLogs) (109627) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:10,362::__init__::232::IOProcess::(_processLogs) (109627) Queuing response >Thread-79::DEBUG::2014-09-25 17:51:10,363::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:51:10,373::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000440911 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:10,375::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:10,375::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:10,375::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:10,375::__init__::232::IOProcess::(_processLogs) (109628) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:10,376::__init__::232::IOProcess::(_processLogs) (109628) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:10,377::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:10,378::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:10,378::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:10,379::__init__::232::IOProcess::(_processLogs) (109629) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:10,379::__init__::232::IOProcess::(_processLogs) (109629) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:10,379::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:10,380::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:10,380::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:10,381::__init__::232::IOProcess::(_processLogs) (109630) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:10,381::__init__::232::IOProcess::(_processLogs) (109630) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:10,381::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:10,382::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:10,383::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:10,383::__init__::232::IOProcess::(_processLogs) (109631) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:10,383::__init__::232::IOProcess::(_processLogs) (109631) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:11,101::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71255::DEBUG::2014-09-25 17:51:11,103::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:51:11,103::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71255::DEBUG::2014-09-25 17:51:11,105::task::595::Storage.TaskManager.Task::(_updateState) Task=`54ad9eb4-069b-41b6-a0c4-dfc5eff8e178`::moving from state init -> state preparing >Thread-71255::INFO::2014-09-25 17:51:11,105::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71255::INFO::2014-09-25 17:51:11,106::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71255::DEBUG::2014-09-25 17:51:11,106::task::1191::Storage.TaskManager.Task::(prepare) Task=`54ad9eb4-069b-41b6-a0c4-dfc5eff8e178`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71255::DEBUG::2014-09-25 17:51:11,106::task::595::Storage.TaskManager.Task::(_updateState) Task=`54ad9eb4-069b-41b6-a0c4-dfc5eff8e178`::moving from state preparing -> state finished >Thread-71255::DEBUG::2014-09-25 17:51:11,107::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71255::DEBUG::2014-09-25 17:51:11,107::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71255::DEBUG::2014-09-25 17:51:11,107::task::993::Storage.TaskManager.Task::(_decref) Task=`54ad9eb4-069b-41b6-a0c4-dfc5eff8e178`::ref 0 aborting False >Thread-71255::DEBUG::2014-09-25 17:51:11,108::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71255::DEBUG::2014-09-25 17:51:11,108::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:11,121::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71256::DEBUG::2014-09-25 17:51:11,123::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:51:11,123::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71256::DEBUG::2014-09-25 17:51:11,125::task::595::Storage.TaskManager.Task::(_updateState) Task=`a9eadf5d-53f9-4448-a036-bfdb48dda479`::moving from state init -> state preparing >Thread-71256::INFO::2014-09-25 17:51:11,125::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71256::DEBUG::2014-09-25 17:51:11,126::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7087cadf-c64e-4f6f-8d45-db005f29d5de`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71256::DEBUG::2014-09-25 17:51:11,126::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71256::DEBUG::2014-09-25 17:51:11,126::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71256::DEBUG::2014-09-25 17:51:11,126::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7087cadf-c64e-4f6f-8d45-db005f29d5de`::Granted request >Thread-71256::DEBUG::2014-09-25 17:51:11,127::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a9eadf5d-53f9-4448-a036-bfdb48dda479`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71256::DEBUG::2014-09-25 17:51:11,127::task::993::Storage.TaskManager.Task::(_decref) Task=`a9eadf5d-53f9-4448-a036-bfdb48dda479`::ref 1 aborting False >Thread-71256::INFO::2014-09-25 17:51:11,132::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71256::DEBUG::2014-09-25 17:51:11,132::task::1191::Storage.TaskManager.Task::(prepare) Task=`a9eadf5d-53f9-4448-a036-bfdb48dda479`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71256::DEBUG::2014-09-25 17:51:11,132::task::595::Storage.TaskManager.Task::(_updateState) Task=`a9eadf5d-53f9-4448-a036-bfdb48dda479`::moving from state preparing -> state finished >Thread-71256::DEBUG::2014-09-25 17:51:11,133::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71256::DEBUG::2014-09-25 17:51:11,133::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71256::DEBUG::2014-09-25 17:51:11,133::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71256::DEBUG::2014-09-25 17:51:11,133::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71256::DEBUG::2014-09-25 17:51:11,134::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71256::DEBUG::2014-09-25 17:51:11,134::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71256::DEBUG::2014-09-25 17:51:11,134::task::993::Storage.TaskManager.Task::(_decref) Task=`a9eadf5d-53f9-4448-a036-bfdb48dda479`::ref 0 aborting False >Thread-71256::DEBUG::2014-09-25 17:51:11,135::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71256::DEBUG::2014-09-25 17:51:11,135::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:12,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:12,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135916 s, 75.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:12,610::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:12,612::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71257::DEBUG::2014-09-25 17:51:12,614::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef9c152e-f9a2-4106-a3d5-42de23ef83ba`::moving from state init -> state preparing >Thread-71257::INFO::2014-09-25 17:51:12,615::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71257::INFO::2014-09-25 17:51:12,615::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000440911', 'lastCheck': '2.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000560617', 'lastCheck': '5.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00480532', 'lastCheck': '7.9', 'valid': True}} >Thread-71257::DEBUG::2014-09-25 17:51:12,615::task::1191::Storage.TaskManager.Task::(prepare) Task=`ef9c152e-f9a2-4106-a3d5-42de23ef83ba`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000440911', 'lastCheck': '2.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000560617', 'lastCheck': '5.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00480532', 'lastCheck': '7.9', 'valid': True}} >Thread-71257::DEBUG::2014-09-25 17:51:12,616::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef9c152e-f9a2-4106-a3d5-42de23ef83ba`::moving from state preparing -> state finished >Thread-71257::DEBUG::2014-09-25 17:51:12,616::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71257::DEBUG::2014-09-25 17:51:12,616::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71257::DEBUG::2014-09-25 17:51:12,616::task::993::Storage.TaskManager.Task::(_decref) Task=`ef9c152e-f9a2-4106-a3d5-42de23ef83ba`::ref 0 aborting False >Thread-71257::DEBUG::2014-09-25 17:51:12,619::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:12,638::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:12,640::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71258::DEBUG::2014-09-25 17:51:12,641::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:14,417::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:14,442::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155717 s, 65.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:14,701::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:14,702::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:14,702::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:14,702::__init__::232::IOProcess::(_processLogs) (109632) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:14,703::__init__::232::IOProcess::(_processLogs) (109632) Queuing response >Thread-81::DEBUG::2014-09-25 17:51:14,704::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:51:14,724::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00614997 s, 60.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:14,725::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:14,725::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:14,726::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:14,726::__init__::232::IOProcess::(_processLogs) (109633) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:14,730::__init__::232::IOProcess::(_processLogs) (109633) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:15,701::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:15,702::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71259::DEBUG::2014-09-25 17:51:15,704::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:16,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:16,543::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0304292 s, 33.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:16,868::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:16,869::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:16,870::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:16,871::__init__::232::IOProcess::(_processLogs) (109634) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:16,873::__init__::232::IOProcess::(_processLogs) (109634) Queuing response >Thread-80::DEBUG::2014-09-25 17:51:16,875::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:51:16,885::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000610388 s, 619 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:16,888::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:16,890::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:16,892::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:16,893::__init__::232::IOProcess::(_processLogs) (109635) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:16,894::__init__::232::IOProcess::(_processLogs) (109635) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:18,719::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:18,720::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71260::DEBUG::2014-09-25 17:51:18,722::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:18,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:18,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132408 s, 77.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:20,383::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:20,384::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:20,384::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:20,384::__init__::232::IOProcess::(_processLogs) (109636) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:20,385::__init__::232::IOProcess::(_processLogs) (109636) Queuing response >Thread-79::DEBUG::2014-09-25 17:51:20,386::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:51:20,395::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000513587 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:20,396::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:20,397::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:20,397::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:20,397::__init__::232::IOProcess::(_processLogs) (109637) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:20,398::__init__::232::IOProcess::(_processLogs) (109637) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:20,399::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:20,400::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:20,400::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:20,400::__init__::232::IOProcess::(_processLogs) (109638) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:20,401::__init__::232::IOProcess::(_processLogs) (109638) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:20,401::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:20,402::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:20,402::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:20,402::__init__::232::IOProcess::(_processLogs) (109639) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:20,403::__init__::232::IOProcess::(_processLogs) (109639) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:20,403::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:20,403::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:20,404::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:20,404::__init__::232::IOProcess::(_processLogs) (109640) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:20,404::__init__::232::IOProcess::(_processLogs) (109640) Queuing response >Dummy-148::DEBUG::2014-09-25 17:51:20,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:20,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134174 s, 76.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:21,163::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71261::DEBUG::2014-09-25 17:51:21,164::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:51:21,164::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71261::DEBUG::2014-09-25 17:51:21,167::task::595::Storage.TaskManager.Task::(_updateState) Task=`04699fad-c82c-4766-9826-3d5898d5e5b7`::moving from state init -> state preparing >Thread-71261::INFO::2014-09-25 17:51:21,167::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71261::INFO::2014-09-25 17:51:21,168::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71261::DEBUG::2014-09-25 17:51:21,168::task::1191::Storage.TaskManager.Task::(prepare) Task=`04699fad-c82c-4766-9826-3d5898d5e5b7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71261::DEBUG::2014-09-25 17:51:21,168::task::595::Storage.TaskManager.Task::(_updateState) Task=`04699fad-c82c-4766-9826-3d5898d5e5b7`::moving from state preparing -> state finished >Thread-71261::DEBUG::2014-09-25 17:51:21,168::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71261::DEBUG::2014-09-25 17:51:21,169::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71261::DEBUG::2014-09-25 17:51:21,169::task::993::Storage.TaskManager.Task::(_decref) Task=`04699fad-c82c-4766-9826-3d5898d5e5b7`::ref 0 aborting False >Thread-71261::DEBUG::2014-09-25 17:51:21,169::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71261::DEBUG::2014-09-25 17:51:21,170::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:21,183::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:21,184::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71262::DEBUG::2014-09-25 17:51:21,184::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71262::DEBUG::2014-09-25 17:51:21,186::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ccca8fd-accb-4ca5-9fb7-e037535241f5`::moving from state init -> state preparing >Thread-71262::INFO::2014-09-25 17:51:21,186::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71262::DEBUG::2014-09-25 17:51:21,187::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`814dd494-96a0-4e13-b3c3-d3574dedc18b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71262::DEBUG::2014-09-25 17:51:21,187::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71262::DEBUG::2014-09-25 17:51:21,188::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71262::DEBUG::2014-09-25 17:51:21,188::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`814dd494-96a0-4e13-b3c3-d3574dedc18b`::Granted request >Thread-71262::DEBUG::2014-09-25 17:51:21,190::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0ccca8fd-accb-4ca5-9fb7-e037535241f5`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71262::DEBUG::2014-09-25 17:51:21,190::task::993::Storage.TaskManager.Task::(_decref) Task=`0ccca8fd-accb-4ca5-9fb7-e037535241f5`::ref 1 aborting False >Thread-71262::INFO::2014-09-25 17:51:21,194::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71262::DEBUG::2014-09-25 17:51:21,194::task::1191::Storage.TaskManager.Task::(prepare) Task=`0ccca8fd-accb-4ca5-9fb7-e037535241f5`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71262::DEBUG::2014-09-25 17:51:21,195::task::595::Storage.TaskManager.Task::(_updateState) Task=`0ccca8fd-accb-4ca5-9fb7-e037535241f5`::moving from state preparing -> state finished >Thread-71262::DEBUG::2014-09-25 17:51:21,195::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71262::DEBUG::2014-09-25 17:51:21,195::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71262::DEBUG::2014-09-25 17:51:21,195::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71262::DEBUG::2014-09-25 17:51:21,196::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71262::DEBUG::2014-09-25 17:51:21,196::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71262::DEBUG::2014-09-25 17:51:21,196::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71262::DEBUG::2014-09-25 17:51:21,197::task::993::Storage.TaskManager.Task::(_decref) Task=`0ccca8fd-accb-4ca5-9fb7-e037535241f5`::ref 0 aborting False >Thread-71262::DEBUG::2014-09-25 17:51:21,197::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71262::DEBUG::2014-09-25 17:51:21,198::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:21,744::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:21,745::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71263::DEBUG::2014-09-25 17:51:21,748::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:22,818::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:22,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133977 s, 76.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:24,732::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:24,732::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:24,733::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:24,733::__init__::232::IOProcess::(_processLogs) (109641) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:24,733::__init__::232::IOProcess::(_processLogs) (109641) Queuing response >Thread-81::DEBUG::2014-09-25 17:51:24,734::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:51:24,743::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000398585 s, 931 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:24,745::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:24,745::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:24,745::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:24,746::__init__::232::IOProcess::(_processLogs) (109642) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:24,746::__init__::232::IOProcess::(_processLogs) (109642) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:24,767::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:24,769::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71264::DEBUG::2014-09-25 17:51:24,771::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:24,859::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:24,881::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136087 s, 75.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:26,893::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:26,894::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:26,894::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:26,894::__init__::232::IOProcess::(_processLogs) (109643) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:26,895::__init__::232::IOProcess::(_processLogs) (109643) Queuing response >Thread-80::DEBUG::2014-09-25 17:51:26,896::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:26,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-80::DEBUG::2014-09-25 17:51:26,905::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000406908 s, 929 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:26,918::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:26,919::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:26,919::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:26,919::__init__::232::IOProcess::(_processLogs) (109644) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:26,920::__init__::232::IOProcess::(_processLogs) (109644) Queuing response >Dummy-148::DEBUG::2014-09-25 17:51:26,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134002 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:27,785::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:27,786::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71265::DEBUG::2014-09-25 17:51:27,789::task::595::Storage.TaskManager.Task::(_updateState) Task=`3d7deba1-6c5f-41e5-9a92-57c1c40b8a6d`::moving from state init -> state preparing >Thread-71265::INFO::2014-09-25 17:51:27,789::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71265::INFO::2014-09-25 17:51:27,790::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000513587', 'lastCheck': '7.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000406908', 'lastCheck': '0.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000398585', 'lastCheck': '3.0', 'valid': True}} >Thread-71265::DEBUG::2014-09-25 17:51:27,790::task::1191::Storage.TaskManager.Task::(prepare) Task=`3d7deba1-6c5f-41e5-9a92-57c1c40b8a6d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000513587', 'lastCheck': '7.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000406908', 'lastCheck': '0.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000398585', 'lastCheck': '3.0', 'valid': True}} >Thread-71265::DEBUG::2014-09-25 17:51:27,790::task::595::Storage.TaskManager.Task::(_updateState) Task=`3d7deba1-6c5f-41e5-9a92-57c1c40b8a6d`::moving from state preparing -> state finished >Thread-71265::DEBUG::2014-09-25 17:51:27,791::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71265::DEBUG::2014-09-25 17:51:27,791::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71265::DEBUG::2014-09-25 17:51:27,791::task::993::Storage.TaskManager.Task::(_decref) Task=`3d7deba1-6c5f-41e5-9a92-57c1c40b8a6d`::ref 0 aborting False >Thread-71265::DEBUG::2014-09-25 17:51:27,793::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:27,812::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:27,814::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71266::DEBUG::2014-09-25 17:51:27,816::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:28,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:28,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131855 s, 77.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:30,406::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:30,406::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:30,407::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:51:30,408::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:51:30,408::__init__::232::IOProcess::(_processLogs) (109645) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:30,417::__init__::232::IOProcess::(_processLogs) (109645) Queuing response >Thread-79::DEBUG::2014-09-25 17:51:30,418::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000517097 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:30,419::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:30,419::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:30,420::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:30,421::__init__::232::IOProcess::(_processLogs) (109646) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:30,421::__init__::232::IOProcess::(_processLogs) (109646) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:30,421::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:30,422::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:30,422::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:30,423::__init__::232::IOProcess::(_processLogs) (109647) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:30,423::__init__::232::IOProcess::(_processLogs) (109647) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:30,423::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:30,424::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:30,424::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:30,425::__init__::232::IOProcess::(_processLogs) (109648) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:30,425::__init__::232::IOProcess::(_processLogs) (109648) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:30,426::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:30,426::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:30,427::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:30,427::__init__::232::IOProcess::(_processLogs) (109649) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:30,427::__init__::232::IOProcess::(_processLogs) (109649) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:30,881::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:30,882::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71267::DEBUG::2014-09-25 17:51:30,884::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:30,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:31,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137462 s, 74.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:31,225::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:31,226::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71268::DEBUG::2014-09-25 17:51:31,226::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71268::DEBUG::2014-09-25 17:51:31,228::task::595::Storage.TaskManager.Task::(_updateState) Task=`a4717271-783a-49a3-93fb-e3782c2f11ca`::moving from state init -> state preparing >Thread-71268::INFO::2014-09-25 17:51:31,229::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71268::INFO::2014-09-25 17:51:31,229::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71268::DEBUG::2014-09-25 17:51:31,230::task::1191::Storage.TaskManager.Task::(prepare) Task=`a4717271-783a-49a3-93fb-e3782c2f11ca`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71268::DEBUG::2014-09-25 17:51:31,230::task::595::Storage.TaskManager.Task::(_updateState) Task=`a4717271-783a-49a3-93fb-e3782c2f11ca`::moving from state preparing -> state finished >Thread-71268::DEBUG::2014-09-25 17:51:31,230::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71268::DEBUG::2014-09-25 17:51:31,230::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71268::DEBUG::2014-09-25 17:51:31,231::task::993::Storage.TaskManager.Task::(_decref) Task=`a4717271-783a-49a3-93fb-e3782c2f11ca`::ref 0 aborting False >Thread-71268::DEBUG::2014-09-25 17:51:31,231::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71268::DEBUG::2014-09-25 17:51:31,231::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:31,245::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:31,246::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71269::DEBUG::2014-09-25 17:51:31,246::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71269::DEBUG::2014-09-25 17:51:31,248::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f0b6923-27de-4e6a-935c-e469ad52bdb4`::moving from state init -> state preparing >Thread-71269::INFO::2014-09-25 17:51:31,248::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71269::DEBUG::2014-09-25 17:51:31,249::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`536adb54-0763-41b2-9b57-05d0f9ed976c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71269::DEBUG::2014-09-25 17:51:31,249::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71269::DEBUG::2014-09-25 17:51:31,250::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71269::DEBUG::2014-09-25 17:51:31,250::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`536adb54-0763-41b2-9b57-05d0f9ed976c`::Granted request >Thread-71269::DEBUG::2014-09-25 17:51:31,250::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0f0b6923-27de-4e6a-935c-e469ad52bdb4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71269::DEBUG::2014-09-25 17:51:31,251::task::993::Storage.TaskManager.Task::(_decref) Task=`0f0b6923-27de-4e6a-935c-e469ad52bdb4`::ref 1 aborting False >Thread-71269::INFO::2014-09-25 17:51:31,255::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71269::DEBUG::2014-09-25 17:51:31,255::task::1191::Storage.TaskManager.Task::(prepare) Task=`0f0b6923-27de-4e6a-935c-e469ad52bdb4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71269::DEBUG::2014-09-25 17:51:31,255::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f0b6923-27de-4e6a-935c-e469ad52bdb4`::moving from state preparing -> state finished >Thread-71269::DEBUG::2014-09-25 17:51:31,256::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71269::DEBUG::2014-09-25 17:51:31,256::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71269::DEBUG::2014-09-25 17:51:31,256::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71269::DEBUG::2014-09-25 17:51:31,257::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71269::DEBUG::2014-09-25 17:51:31,257::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71269::DEBUG::2014-09-25 17:51:31,257::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71269::DEBUG::2014-09-25 17:51:31,257::task::993::Storage.TaskManager.Task::(_decref) Task=`0f0b6923-27de-4e6a-935c-e469ad52bdb4`::ref 0 aborting False >Thread-71269::DEBUG::2014-09-25 17:51:31,258::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71269::DEBUG::2014-09-25 17:51:31,258::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:33,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:33,053::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135153 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:33,900::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:33,901::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71270::DEBUG::2014-09-25 17:51:33,903::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:51:34,748::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:34,749::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:34,749::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:34,749::__init__::232::IOProcess::(_processLogs) (109650) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:34,749::__init__::232::IOProcess::(_processLogs) (109650) Queuing response >Thread-81::DEBUG::2014-09-25 17:51:34,751::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:51:34,760::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000413281 s, 898 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:34,761::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:34,761::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:34,761::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:34,762::__init__::232::IOProcess::(_processLogs) (109651) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:34,762::__init__::232::IOProcess::(_processLogs) (109651) Queuing response >Dummy-148::DEBUG::2014-09-25 17:51:35,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:35,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137172 s, 74.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:36,919::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:36,921::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-29::DEBUG::2014-09-25 17:51:36,924::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:36,925::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:36,926::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-71271::DEBUG::2014-09-25 17:51:36,927::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:51:36,927::__init__::232::IOProcess::(_processLogs) (109652) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:36,930::__init__::232::IOProcess::(_processLogs) (109652) Queuing response >Thread-80::DEBUG::2014-09-25 17:51:36,932::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:51:36,940::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000450792 s, 839 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:36,941::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:36,942::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:36,942::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:36,942::__init__::232::IOProcess::(_processLogs) (109653) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:36,943::__init__::232::IOProcess::(_processLogs) (109653) Queuing response >Dummy-148::DEBUG::2014-09-25 17:51:37,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:37,136::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135423 s, 75.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:51:39,156::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:39,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164037 s, 62.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:39,946::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:39,948::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71272::DEBUG::2014-09-25 17:51:39,950::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:51:40,428::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:40,428::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:40,428::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:40,429::__init__::232::IOProcess::(_processLogs) (109654) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:40,429::__init__::232::IOProcess::(_processLogs) (109654) Queuing response >Thread-79::DEBUG::2014-09-25 17:51:40,430::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:51:40,439::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00042979 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:40,440::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:40,441::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:40,441::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:40,441::__init__::232::IOProcess::(_processLogs) (109655) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:40,442::__init__::232::IOProcess::(_processLogs) (109655) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:40,444::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:40,444::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:40,444::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:40,445::__init__::232::IOProcess::(_processLogs) (109656) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:40,445::__init__::232::IOProcess::(_processLogs) (109656) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:40,446::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:40,446::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:40,447::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:40,447::__init__::232::IOProcess::(_processLogs) (109657) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:40,447::__init__::232::IOProcess::(_processLogs) (109657) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:40,448::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:40,448::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:40,449::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:40,450::__init__::232::IOProcess::(_processLogs) (109658) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:40,450::__init__::232::IOProcess::(_processLogs) (109658) Queuing response >Dummy-148::DEBUG::2014-09-25 17:51:41,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:41,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013159 s, 77.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:41,288::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:41,290::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71273::DEBUG::2014-09-25 17:51:41,290::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71273::DEBUG::2014-09-25 17:51:41,292::task::595::Storage.TaskManager.Task::(_updateState) Task=`4814cd0a-1beb-4a99-a50e-293f61fbf8eb`::moving from state init -> state preparing >Thread-71273::INFO::2014-09-25 17:51:41,292::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71273::INFO::2014-09-25 17:51:41,293::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71273::DEBUG::2014-09-25 17:51:41,293::task::1191::Storage.TaskManager.Task::(prepare) Task=`4814cd0a-1beb-4a99-a50e-293f61fbf8eb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71273::DEBUG::2014-09-25 17:51:41,294::task::595::Storage.TaskManager.Task::(_updateState) Task=`4814cd0a-1beb-4a99-a50e-293f61fbf8eb`::moving from state preparing -> state finished >Thread-71273::DEBUG::2014-09-25 17:51:41,294::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71273::DEBUG::2014-09-25 17:51:41,294::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71273::DEBUG::2014-09-25 17:51:41,294::task::993::Storage.TaskManager.Task::(_decref) Task=`4814cd0a-1beb-4a99-a50e-293f61fbf8eb`::ref 0 aborting False >Thread-71273::DEBUG::2014-09-25 17:51:41,295::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71273::DEBUG::2014-09-25 17:51:41,295::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:41,308::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:41,309::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71274::DEBUG::2014-09-25 17:51:41,310::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71274::DEBUG::2014-09-25 17:51:41,311::task::595::Storage.TaskManager.Task::(_updateState) Task=`b0f9d673-9d3c-4a34-ad15-a5758553b468`::moving from state init -> state preparing >Thread-71274::INFO::2014-09-25 17:51:41,311::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71274::DEBUG::2014-09-25 17:51:41,312::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`199899d1-8314-4567-ae83-9f33fdc8c6ab`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71274::DEBUG::2014-09-25 17:51:41,312::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71274::DEBUG::2014-09-25 17:51:41,313::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71274::DEBUG::2014-09-25 17:51:41,313::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`199899d1-8314-4567-ae83-9f33fdc8c6ab`::Granted request >Thread-71274::DEBUG::2014-09-25 17:51:41,314::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b0f9d673-9d3c-4a34-ad15-a5758553b468`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71274::DEBUG::2014-09-25 17:51:41,314::task::993::Storage.TaskManager.Task::(_decref) Task=`b0f9d673-9d3c-4a34-ad15-a5758553b468`::ref 1 aborting False >Thread-71274::INFO::2014-09-25 17:51:41,318::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71274::DEBUG::2014-09-25 17:51:41,319::task::1191::Storage.TaskManager.Task::(prepare) Task=`b0f9d673-9d3c-4a34-ad15-a5758553b468`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71274::DEBUG::2014-09-25 17:51:41,319::task::595::Storage.TaskManager.Task::(_updateState) Task=`b0f9d673-9d3c-4a34-ad15-a5758553b468`::moving from state preparing -> state finished >Thread-71274::DEBUG::2014-09-25 17:51:41,319::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71274::DEBUG::2014-09-25 17:51:41,319::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71274::DEBUG::2014-09-25 17:51:41,320::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71274::DEBUG::2014-09-25 17:51:41,320::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71274::DEBUG::2014-09-25 17:51:41,320::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71274::DEBUG::2014-09-25 17:51:41,320::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71274::DEBUG::2014-09-25 17:51:41,321::task::993::Storage.TaskManager.Task::(_decref) Task=`b0f9d673-9d3c-4a34-ad15-a5758553b468`::ref 0 aborting False >Thread-71274::DEBUG::2014-09-25 17:51:41,321::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71274::DEBUG::2014-09-25 17:51:41,322::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:42,964::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:42,966::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71275::DEBUG::2014-09-25 17:51:42,968::task::595::Storage.TaskManager.Task::(_updateState) Task=`655e8e6e-dbef-4b3a-9e7d-a8eb23e8cf1b`::moving from state init -> state preparing >Thread-71275::INFO::2014-09-25 17:51:42,969::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71275::INFO::2014-09-25 17:51:42,969::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00042979', 'lastCheck': '2.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000450792', 'lastCheck': '6.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000413281', 'lastCheck': '8.2', 'valid': True}} >Thread-71275::DEBUG::2014-09-25 17:51:42,970::task::1191::Storage.TaskManager.Task::(prepare) Task=`655e8e6e-dbef-4b3a-9e7d-a8eb23e8cf1b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00042979', 'lastCheck': '2.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000450792', 'lastCheck': '6.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000413281', 'lastCheck': '8.2', 'valid': True}} >Thread-71275::DEBUG::2014-09-25 17:51:42,970::task::595::Storage.TaskManager.Task::(_updateState) Task=`655e8e6e-dbef-4b3a-9e7d-a8eb23e8cf1b`::moving from state preparing -> state finished >Thread-71275::DEBUG::2014-09-25 17:51:42,970::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71275::DEBUG::2014-09-25 17:51:42,970::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71275::DEBUG::2014-09-25 17:51:42,970::task::993::Storage.TaskManager.Task::(_decref) Task=`655e8e6e-dbef-4b3a-9e7d-a8eb23e8cf1b`::ref 0 aborting False >Thread-71275::DEBUG::2014-09-25 17:51:42,973::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:42,996::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:42,997::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71276::DEBUG::2014-09-25 17:51:42,999::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:43,245::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:43,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.219001 s, 4.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:44,764::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:44,765::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:44,765::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:44,765::__init__::232::IOProcess::(_processLogs) (109659) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:44,768::__init__::232::IOProcess::(_processLogs) (109659) Queuing response >Thread-81::DEBUG::2014-09-25 17:51:44,769::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:51:44,783::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0040874 s, 90.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:44,784::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:44,784::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:44,785::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:44,785::__init__::232::IOProcess::(_processLogs) (109660) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:44,786::__init__::232::IOProcess::(_processLogs) (109660) Queuing response >Dummy-148::DEBUG::2014-09-25 17:51:45,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:45,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0224445 s, 45.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:46,061::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:46,065::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71277::DEBUG::2014-09-25 17:51:46,070::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:51:46,946::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:46,948::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:46,949::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:46,950::__init__::232::IOProcess::(_processLogs) (109661) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:46,952::__init__::232::IOProcess::(_processLogs) (109661) Queuing response >Thread-80::DEBUG::2014-09-25 17:51:46,954::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:51:46,965::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000470556 s, 803 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:46,968::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:46,969::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:46,970::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:46,971::__init__::232::IOProcess::(_processLogs) (109662) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:46,973::__init__::232::IOProcess::(_processLogs) (109662) Queuing response >Dummy-148::DEBUG::2014-09-25 17:51:47,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:47,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185959 s, 55.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:49,087::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:49,089::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71278::DEBUG::2014-09-25 17:51:49,091::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:49,599::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:49,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130725 s, 78.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:50,450::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:50,451::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:50,451::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:50,451::__init__::232::IOProcess::(_processLogs) (109663) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:50,452::__init__::232::IOProcess::(_processLogs) (109663) Queuing response >Thread-79::DEBUG::2014-09-25 17:51:50,453::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:51:50,462::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000461349 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:50,463::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:50,463::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:50,463::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:50,464::__init__::232::IOProcess::(_processLogs) (109664) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:50,464::__init__::232::IOProcess::(_processLogs) (109664) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:50,465::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:50,466::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:50,466::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:50,466::__init__::232::IOProcess::(_processLogs) (109665) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:50,467::__init__::232::IOProcess::(_processLogs) (109665) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:50,468::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:50,468::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:50,468::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:50,469::__init__::232::IOProcess::(_processLogs) (109666) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:50,469::__init__::232::IOProcess::(_processLogs) (109666) Queuing response >Thread-29::DEBUG::2014-09-25 17:51:50,470::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:50,470::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:50,470::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:50,471::__init__::232::IOProcess::(_processLogs) (109667) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:51:50,471::__init__::232::IOProcess::(_processLogs) (109667) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:51,351::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71279::DEBUG::2014-09-25 17:51:51,352::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:51:51,352::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71279::DEBUG::2014-09-25 17:51:51,355::task::595::Storage.TaskManager.Task::(_updateState) Task=`edd908fa-bf8b-420a-876e-c95a51784332`::moving from state init -> state preparing >Thread-71279::INFO::2014-09-25 17:51:51,355::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71279::INFO::2014-09-25 17:51:51,356::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71279::DEBUG::2014-09-25 17:51:51,356::task::1191::Storage.TaskManager.Task::(prepare) Task=`edd908fa-bf8b-420a-876e-c95a51784332`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71279::DEBUG::2014-09-25 17:51:51,356::task::595::Storage.TaskManager.Task::(_updateState) Task=`edd908fa-bf8b-420a-876e-c95a51784332`::moving from state preparing -> state finished >Thread-71279::DEBUG::2014-09-25 17:51:51,356::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71279::DEBUG::2014-09-25 17:51:51,356::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71279::DEBUG::2014-09-25 17:51:51,357::task::993::Storage.TaskManager.Task::(_decref) Task=`edd908fa-bf8b-420a-876e-c95a51784332`::ref 0 aborting False >Thread-71279::DEBUG::2014-09-25 17:51:51,357::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71279::DEBUG::2014-09-25 17:51:51,357::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:51,371::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:51,373::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71280::DEBUG::2014-09-25 17:51:51,373::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71280::DEBUG::2014-09-25 17:51:51,374::task::595::Storage.TaskManager.Task::(_updateState) Task=`590357b1-db13-4456-9e5b-ef2396f1a697`::moving from state init -> state preparing >Thread-71280::INFO::2014-09-25 17:51:51,375::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71280::DEBUG::2014-09-25 17:51:51,375::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2afeb7b0-7e71-4c3d-b1ec-7eb141e1f8dc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71280::DEBUG::2014-09-25 17:51:51,376::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71280::DEBUG::2014-09-25 17:51:51,376::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71280::DEBUG::2014-09-25 17:51:51,376::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2afeb7b0-7e71-4c3d-b1ec-7eb141e1f8dc`::Granted request >Thread-71280::DEBUG::2014-09-25 17:51:51,377::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`590357b1-db13-4456-9e5b-ef2396f1a697`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71280::DEBUG::2014-09-25 17:51:51,377::task::993::Storage.TaskManager.Task::(_decref) Task=`590357b1-db13-4456-9e5b-ef2396f1a697`::ref 1 aborting False >Thread-71280::INFO::2014-09-25 17:51:51,381::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71280::DEBUG::2014-09-25 17:51:51,382::task::1191::Storage.TaskManager.Task::(prepare) Task=`590357b1-db13-4456-9e5b-ef2396f1a697`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71280::DEBUG::2014-09-25 17:51:51,382::task::595::Storage.TaskManager.Task::(_updateState) Task=`590357b1-db13-4456-9e5b-ef2396f1a697`::moving from state preparing -> state finished >Thread-71280::DEBUG::2014-09-25 17:51:51,382::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71280::DEBUG::2014-09-25 17:51:51,382::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71280::DEBUG::2014-09-25 17:51:51,383::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71280::DEBUG::2014-09-25 17:51:51,383::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71280::DEBUG::2014-09-25 17:51:51,383::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71280::DEBUG::2014-09-25 17:51:51,384::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71280::DEBUG::2014-09-25 17:51:51,384::task::993::Storage.TaskManager.Task::(_decref) Task=`590357b1-db13-4456-9e5b-ef2396f1a697`::ref 0 aborting False >Thread-71280::DEBUG::2014-09-25 17:51:51,384::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71280::DEBUG::2014-09-25 17:51:51,385::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:51,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:51,663::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132822 s, 77.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:52,106::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:52,108::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71281::DEBUG::2014-09-25 17:51:52,110::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:53,683::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:53,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135121 s, 75.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:54,788::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:54,788::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:54,788::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:54,789::__init__::232::IOProcess::(_processLogs) (109668) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:54,789::__init__::232::IOProcess::(_processLogs) (109668) Queuing response >Thread-81::DEBUG::2014-09-25 17:51:54,791::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:51:54,799::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000443253 s, 837 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:54,801::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:54,801::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:54,801::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:54,802::__init__::232::IOProcess::(_processLogs) (109669) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:54,803::__init__::232::IOProcess::(_processLogs) (109669) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:55,124::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:55,126::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71282::DEBUG::2014-09-25 17:51:55,128::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:55,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:55,747::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132357 s, 77.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:56,976::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:56,977::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:56,977::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:56,977::__init__::232::IOProcess::(_processLogs) (109670) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:56,978::__init__::232::IOProcess::(_processLogs) (109670) Queuing response >Thread-80::DEBUG::2014-09-25 17:51:56,979::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:51:56,988::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000411582 s, 918 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:51:56,989::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:51:56,989::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:51:56,990::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:51:56,991::__init__::232::IOProcess::(_processLogs) (109671) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:51:56,991::__init__::232::IOProcess::(_processLogs) (109671) Queuing response >Dummy-148::DEBUG::2014-09-25 17:51:57,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:57,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135009 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:58,138::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:58,139::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71283::DEBUG::2014-09-25 17:51:58,142::task::595::Storage.TaskManager.Task::(_updateState) Task=`9d56dbc5-8563-40a6-b4dd-bc4d66590ef0`::moving from state init -> state preparing >Thread-71283::INFO::2014-09-25 17:51:58,143::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71283::INFO::2014-09-25 17:51:58,143::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000461349', 'lastCheck': '7.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000411582', 'lastCheck': '1.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443253', 'lastCheck': '3.3', 'valid': True}} >Thread-71283::DEBUG::2014-09-25 17:51:58,144::task::1191::Storage.TaskManager.Task::(prepare) Task=`9d56dbc5-8563-40a6-b4dd-bc4d66590ef0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000461349', 'lastCheck': '7.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000411582', 'lastCheck': '1.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443253', 'lastCheck': '3.3', 'valid': True}} >Thread-71283::DEBUG::2014-09-25 17:51:58,144::task::595::Storage.TaskManager.Task::(_updateState) Task=`9d56dbc5-8563-40a6-b4dd-bc4d66590ef0`::moving from state preparing -> state finished >Thread-71283::DEBUG::2014-09-25 17:51:58,144::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71283::DEBUG::2014-09-25 17:51:58,144::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71283::DEBUG::2014-09-25 17:51:58,144::task::993::Storage.TaskManager.Task::(_decref) Task=`9d56dbc5-8563-40a6-b4dd-bc4d66590ef0`::ref 0 aborting False >Thread-71283::DEBUG::2014-09-25 17:51:58,147::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:51:58,164::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:51:58,165::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71284::DEBUG::2014-09-25 17:51:58,167::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:51:59,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:51:59,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149165 s, 68.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:00,473::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:00,473::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:00,473::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:00,474::__init__::232::IOProcess::(_processLogs) (109672) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:00,475::__init__::232::IOProcess::(_processLogs) (109672) Queuing response >Thread-79::DEBUG::2014-09-25 17:52:00,476::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:52:00,493::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00242418 s, 232 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:00,494::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:00,494::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:00,495::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:00,495::__init__::232::IOProcess::(_processLogs) (109673) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:00,496::__init__::232::IOProcess::(_processLogs) (109673) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:00,497::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:00,498::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:00,498::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:00,498::__init__::232::IOProcess::(_processLogs) (109674) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:00,498::__init__::232::IOProcess::(_processLogs) (109674) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:00,499::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:00,500::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:00,501::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:00,501::__init__::232::IOProcess::(_processLogs) (109675) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:00,501::__init__::232::IOProcess::(_processLogs) (109675) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:00,502::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:00,502::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:00,503::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:00,503::__init__::232::IOProcess::(_processLogs) (109676) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:00,503::__init__::232::IOProcess::(_processLogs) (109676) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:01,212::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:01,214::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71285::DEBUG::2014-09-25 17:52:01,216::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:01,412::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71286::DEBUG::2014-09-25 17:52:01,413::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:52:01,413::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71286::DEBUG::2014-09-25 17:52:01,415::task::595::Storage.TaskManager.Task::(_updateState) Task=`70c4d2b0-9a5a-4766-a8e2-e98d851b3ba0`::moving from state init -> state preparing >Thread-71286::INFO::2014-09-25 17:52:01,415::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71286::INFO::2014-09-25 17:52:01,416::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71286::DEBUG::2014-09-25 17:52:01,416::task::1191::Storage.TaskManager.Task::(prepare) Task=`70c4d2b0-9a5a-4766-a8e2-e98d851b3ba0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71286::DEBUG::2014-09-25 17:52:01,416::task::595::Storage.TaskManager.Task::(_updateState) Task=`70c4d2b0-9a5a-4766-a8e2-e98d851b3ba0`::moving from state preparing -> state finished >Thread-71286::DEBUG::2014-09-25 17:52:01,417::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71286::DEBUG::2014-09-25 17:52:01,417::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71286::DEBUG::2014-09-25 17:52:01,417::task::993::Storage.TaskManager.Task::(_decref) Task=`70c4d2b0-9a5a-4766-a8e2-e98d851b3ba0`::ref 0 aborting False >Thread-71286::DEBUG::2014-09-25 17:52:01,417::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71286::DEBUG::2014-09-25 17:52:01,418::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:01,428::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71287::DEBUG::2014-09-25 17:52:01,430::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:52:01,430::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71287::DEBUG::2014-09-25 17:52:01,432::task::595::Storage.TaskManager.Task::(_updateState) Task=`24b76dd0-6b03-4364-ad7a-a1b32edde727`::moving from state init -> state preparing >Thread-71287::INFO::2014-09-25 17:52:01,432::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71287::DEBUG::2014-09-25 17:52:01,433::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`538cf470-21a7-4253-a889-f6a10cd1d19f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71287::DEBUG::2014-09-25 17:52:01,433::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71287::DEBUG::2014-09-25 17:52:01,433::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71287::DEBUG::2014-09-25 17:52:01,433::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`538cf470-21a7-4253-a889-f6a10cd1d19f`::Granted request >Thread-71287::DEBUG::2014-09-25 17:52:01,435::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`24b76dd0-6b03-4364-ad7a-a1b32edde727`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71287::DEBUG::2014-09-25 17:52:01,435::task::993::Storage.TaskManager.Task::(_decref) Task=`24b76dd0-6b03-4364-ad7a-a1b32edde727`::ref 1 aborting False >Thread-71287::INFO::2014-09-25 17:52:01,440::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71287::DEBUG::2014-09-25 17:52:01,440::task::1191::Storage.TaskManager.Task::(prepare) Task=`24b76dd0-6b03-4364-ad7a-a1b32edde727`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71287::DEBUG::2014-09-25 17:52:01,440::task::595::Storage.TaskManager.Task::(_updateState) Task=`24b76dd0-6b03-4364-ad7a-a1b32edde727`::moving from state preparing -> state finished >Thread-71287::DEBUG::2014-09-25 17:52:01,440::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71287::DEBUG::2014-09-25 17:52:01,441::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71287::DEBUG::2014-09-25 17:52:01,441::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71287::DEBUG::2014-09-25 17:52:01,441::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71287::DEBUG::2014-09-25 17:52:01,441::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71287::DEBUG::2014-09-25 17:52:01,442::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71287::DEBUG::2014-09-25 17:52:01,442::task::993::Storage.TaskManager.Task::(_decref) Task=`24b76dd0-6b03-4364-ad7a-a1b32edde727`::ref 0 aborting False >Thread-71287::DEBUG::2014-09-25 17:52:01,442::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71287::DEBUG::2014-09-25 17:52:01,443::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:01,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:01,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133668 s, 76.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:52:03,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:03,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135371 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:04,227::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:04,228::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71288::DEBUG::2014-09-25 17:52:04,230::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:52:04,804::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:04,805::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:04,805::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:04,805::__init__::232::IOProcess::(_processLogs) (109677) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:04,805::__init__::232::IOProcess::(_processLogs) (109677) Queuing response >Thread-81::DEBUG::2014-09-25 17:52:04,807::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:52:04,816::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000318675 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:04,817::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:04,817::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:04,818::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:04,818::__init__::232::IOProcess::(_processLogs) (109678) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:04,818::__init__::232::IOProcess::(_processLogs) (109678) Queuing response >Dummy-148::DEBUG::2014-09-25 17:52:05,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:05,967::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162137 s, 63.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:06,992::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:06,993::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:06,993::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:06,993::__init__::232::IOProcess::(_processLogs) (109679) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:06,993::__init__::232::IOProcess::(_processLogs) (109679) Queuing response >Thread-80::DEBUG::2014-09-25 17:52:06,995::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:52:07,004::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000588831 s, 642 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:07,005::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:07,005::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:07,006::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:07,006::__init__::232::IOProcess::(_processLogs) (109680) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:07,007::__init__::232::IOProcess::(_processLogs) (109680) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:07,244::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:07,246::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71289::DEBUG::2014-09-25 17:52:07,248::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:07,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:08,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132435 s, 77.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:52:10,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:10,050::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134785 s, 76.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:10,265::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:10,267::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71290::DEBUG::2014-09-25 17:52:10,269::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:52:10,504::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:10,504::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:10,504::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:10,506::__init__::232::IOProcess::(_processLogs) (109681) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 17:52:10,506::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:52:10,507::__init__::232::IOProcess::(_processLogs) (109681) Queuing response >Thread-79::DEBUG::2014-09-25 17:52:10,516::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000368667 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:10,517::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:10,518::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:10,518::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:10,519::__init__::232::IOProcess::(_processLogs) (109682) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:10,520::__init__::232::IOProcess::(_processLogs) (109682) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:10,520::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:10,520::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:10,521::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:10,522::__init__::232::IOProcess::(_processLogs) (109683) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:10,522::__init__::232::IOProcess::(_processLogs) (109683) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:10,522::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:10,523::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:10,523::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:10,524::__init__::232::IOProcess::(_processLogs) (109684) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:10,524::__init__::232::IOProcess::(_processLogs) (109684) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:10,524::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:10,525::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:10,525::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:10,526::__init__::232::IOProcess::(_processLogs) (109685) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:10,526::__init__::232::IOProcess::(_processLogs) (109685) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:11,469::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71291::DEBUG::2014-09-25 17:52:11,470::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:52:11,470::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71291::DEBUG::2014-09-25 17:52:11,473::task::595::Storage.TaskManager.Task::(_updateState) Task=`68945e21-6f35-45a6-b80a-d8a1d82426aa`::moving from state init -> state preparing >Thread-71291::INFO::2014-09-25 17:52:11,473::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71291::INFO::2014-09-25 17:52:11,473::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71291::DEBUG::2014-09-25 17:52:11,474::task::1191::Storage.TaskManager.Task::(prepare) Task=`68945e21-6f35-45a6-b80a-d8a1d82426aa`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71291::DEBUG::2014-09-25 17:52:11,474::task::595::Storage.TaskManager.Task::(_updateState) Task=`68945e21-6f35-45a6-b80a-d8a1d82426aa`::moving from state preparing -> state finished >Thread-71291::DEBUG::2014-09-25 17:52:11,474::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71291::DEBUG::2014-09-25 17:52:11,474::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71291::DEBUG::2014-09-25 17:52:11,475::task::993::Storage.TaskManager.Task::(_decref) Task=`68945e21-6f35-45a6-b80a-d8a1d82426aa`::ref 0 aborting False >Thread-71291::DEBUG::2014-09-25 17:52:11,475::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71291::DEBUG::2014-09-25 17:52:11,475::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:11,530::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71292::DEBUG::2014-09-25 17:52:11,534::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:52:11,534::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71292::DEBUG::2014-09-25 17:52:11,539::task::595::Storage.TaskManager.Task::(_updateState) Task=`547fd8de-bed5-4686-a7b1-a90531b8b56b`::moving from state init -> state preparing >Thread-71292::INFO::2014-09-25 17:52:11,540::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71292::DEBUG::2014-09-25 17:52:11,543::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ca595f56-46b1-4551-a375-2f55709356a2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71292::DEBUG::2014-09-25 17:52:11,544::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71292::DEBUG::2014-09-25 17:52:11,546::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71292::DEBUG::2014-09-25 17:52:11,546::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ca595f56-46b1-4551-a375-2f55709356a2`::Granted request >Thread-71292::DEBUG::2014-09-25 17:52:11,547::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`547fd8de-bed5-4686-a7b1-a90531b8b56b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71292::DEBUG::2014-09-25 17:52:11,547::task::993::Storage.TaskManager.Task::(_decref) Task=`547fd8de-bed5-4686-a7b1-a90531b8b56b`::ref 1 aborting False >Thread-71292::INFO::2014-09-25 17:52:11,551::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71292::DEBUG::2014-09-25 17:52:11,552::task::1191::Storage.TaskManager.Task::(prepare) Task=`547fd8de-bed5-4686-a7b1-a90531b8b56b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71292::DEBUG::2014-09-25 17:52:11,552::task::595::Storage.TaskManager.Task::(_updateState) Task=`547fd8de-bed5-4686-a7b1-a90531b8b56b`::moving from state preparing -> state finished >Thread-71292::DEBUG::2014-09-25 17:52:11,552::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71292::DEBUG::2014-09-25 17:52:11,552::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71292::DEBUG::2014-09-25 17:52:11,553::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71292::DEBUG::2014-09-25 17:52:11,553::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71292::DEBUG::2014-09-25 17:52:11,553::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71292::DEBUG::2014-09-25 17:52:11,553::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71292::DEBUG::2014-09-25 17:52:11,554::task::993::Storage.TaskManager.Task::(_decref) Task=`547fd8de-bed5-4686-a7b1-a90531b8b56b`::ref 0 aborting False >Thread-71292::DEBUG::2014-09-25 17:52:11,554::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71292::DEBUG::2014-09-25 17:52:11,555::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:12,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:12,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015469 s, 66.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:13,287::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:13,289::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71293::DEBUG::2014-09-25 17:52:13,292::task::595::Storage.TaskManager.Task::(_updateState) Task=`da3b385f-ce96-4668-beb6-95ea2dd3ce39`::moving from state init -> state preparing >Thread-71293::INFO::2014-09-25 17:52:13,292::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71293::INFO::2014-09-25 17:52:13,293::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000368667', 'lastCheck': '2.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000588831', 'lastCheck': '6.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000318675', 'lastCheck': '8.5', 'valid': True}} >Thread-71293::DEBUG::2014-09-25 17:52:13,293::task::1191::Storage.TaskManager.Task::(prepare) Task=`da3b385f-ce96-4668-beb6-95ea2dd3ce39`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000368667', 'lastCheck': '2.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000588831', 'lastCheck': '6.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000318675', 'lastCheck': '8.5', 'valid': True}} >Thread-71293::DEBUG::2014-09-25 17:52:13,293::task::595::Storage.TaskManager.Task::(_updateState) Task=`da3b385f-ce96-4668-beb6-95ea2dd3ce39`::moving from state preparing -> state finished >Thread-71293::DEBUG::2014-09-25 17:52:13,293::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71293::DEBUG::2014-09-25 17:52:13,293::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71293::DEBUG::2014-09-25 17:52:13,294::task::993::Storage.TaskManager.Task::(_decref) Task=`da3b385f-ce96-4668-beb6-95ea2dd3ce39`::ref 0 aborting False >Thread-71293::DEBUG::2014-09-25 17:52:13,296::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:13,318::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:13,320::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71294::DEBUG::2014-09-25 17:52:13,322::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:14,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:14,137::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132053 s, 77.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:14,820::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:14,820::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:14,820::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:14,821::__init__::232::IOProcess::(_processLogs) (109686) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:14,821::__init__::232::IOProcess::(_processLogs) (109686) Queuing response >Thread-81::DEBUG::2014-09-25 17:52:14,822::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:52:14,831::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000429998 s, 863 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:14,832::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:14,833::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:14,833::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:14,833::__init__::232::IOProcess::(_processLogs) (109687) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:14,834::__init__::232::IOProcess::(_processLogs) (109687) Queuing response >Dummy-148::DEBUG::2014-09-25 17:52:16,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:16,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0228703 s, 44.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:16,390::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:16,396::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71295::DEBUG::2014-09-25 17:52:16,402::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:52:17,009::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:17,010::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:17,011::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:52:17,013::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:52:17,013::__init__::232::IOProcess::(_processLogs) (109688) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:17,019::__init__::232::IOProcess::(_processLogs) (109688) Queuing response >Thread-80::DEBUG::2014-09-25 17:52:17,020::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000475917 s, 794 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:17,021::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:17,021::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:17,022::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:17,022::__init__::232::IOProcess::(_processLogs) (109689) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:17,022::__init__::232::IOProcess::(_processLogs) (109689) Queuing response >Dummy-148::DEBUG::2014-09-25 17:52:18,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:18,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162931 s, 62.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:19,422::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:19,423::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71296::DEBUG::2014-09-25 17:52:19,426::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:20,451::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:20,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130191 s, 78.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:20,527::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:20,527::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:20,528::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:20,528::__init__::232::IOProcess::(_processLogs) (109690) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:20,528::__init__::232::IOProcess::(_processLogs) (109690) Queuing response >Thread-79::DEBUG::2014-09-25 17:52:20,530::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:52:20,539::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000534334 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:20,540::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:20,540::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:20,540::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:20,541::__init__::232::IOProcess::(_processLogs) (109691) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:20,541::__init__::232::IOProcess::(_processLogs) (109691) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:20,543::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:20,543::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:20,544::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:20,544::__init__::232::IOProcess::(_processLogs) (109692) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:20,545::__init__::232::IOProcess::(_processLogs) (109692) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:20,545::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:20,545::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:20,546::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:20,546::__init__::232::IOProcess::(_processLogs) (109693) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:20,547::__init__::232::IOProcess::(_processLogs) (109693) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:20,547::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:20,547::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:20,548::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:20,549::__init__::232::IOProcess::(_processLogs) (109694) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:20,549::__init__::232::IOProcess::(_processLogs) (109694) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:21,715::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:21,718::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71297::DEBUG::2014-09-25 17:52:21,718::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71297::DEBUG::2014-09-25 17:52:21,721::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c68e5f1-e94c-4c02-a69f-89b3c091659e`::moving from state init -> state preparing >Thread-71297::INFO::2014-09-25 17:52:21,721::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71297::INFO::2014-09-25 17:52:21,722::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71297::DEBUG::2014-09-25 17:52:21,722::task::1191::Storage.TaskManager.Task::(prepare) Task=`1c68e5f1-e94c-4c02-a69f-89b3c091659e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71297::DEBUG::2014-09-25 17:52:21,722::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c68e5f1-e94c-4c02-a69f-89b3c091659e`::moving from state preparing -> state finished >Thread-71297::DEBUG::2014-09-25 17:52:21,722::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71297::DEBUG::2014-09-25 17:52:21,723::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71297::DEBUG::2014-09-25 17:52:21,723::task::993::Storage.TaskManager.Task::(_decref) Task=`1c68e5f1-e94c-4c02-a69f-89b3c091659e`::ref 0 aborting False >Thread-71297::DEBUG::2014-09-25 17:52:21,723::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71297::DEBUG::2014-09-25 17:52:21,724::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:21,739::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71298::DEBUG::2014-09-25 17:52:21,740::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:52:21,740::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71298::DEBUG::2014-09-25 17:52:21,742::task::595::Storage.TaskManager.Task::(_updateState) Task=`951166ed-d523-4954-b46a-27e4847e05e2`::moving from state init -> state preparing >Thread-71298::INFO::2014-09-25 17:52:21,742::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71298::DEBUG::2014-09-25 17:52:21,743::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`944ff9db-5e79-4974-876d-ef0a6394729e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71298::DEBUG::2014-09-25 17:52:21,743::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71298::DEBUG::2014-09-25 17:52:21,744::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71298::DEBUG::2014-09-25 17:52:21,744::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`944ff9db-5e79-4974-876d-ef0a6394729e`::Granted request >Thread-71298::DEBUG::2014-09-25 17:52:21,744::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`951166ed-d523-4954-b46a-27e4847e05e2`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71298::DEBUG::2014-09-25 17:52:21,745::task::993::Storage.TaskManager.Task::(_decref) Task=`951166ed-d523-4954-b46a-27e4847e05e2`::ref 1 aborting False >Thread-71298::INFO::2014-09-25 17:52:21,749::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71298::DEBUG::2014-09-25 17:52:21,749::task::1191::Storage.TaskManager.Task::(prepare) Task=`951166ed-d523-4954-b46a-27e4847e05e2`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71298::DEBUG::2014-09-25 17:52:21,750::task::595::Storage.TaskManager.Task::(_updateState) Task=`951166ed-d523-4954-b46a-27e4847e05e2`::moving from state preparing -> state finished >Thread-71298::DEBUG::2014-09-25 17:52:21,750::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71298::DEBUG::2014-09-25 17:52:21,750::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71298::DEBUG::2014-09-25 17:52:21,750::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71298::DEBUG::2014-09-25 17:52:21,751::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71298::DEBUG::2014-09-25 17:52:21,751::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71298::DEBUG::2014-09-25 17:52:21,751::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71298::DEBUG::2014-09-25 17:52:21,752::task::993::Storage.TaskManager.Task::(_decref) Task=`951166ed-d523-4954-b46a-27e4847e05e2`::ref 0 aborting False >Thread-71298::DEBUG::2014-09-25 17:52:21,752::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71298::DEBUG::2014-09-25 17:52:21,752::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:22,443::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:22,444::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71299::DEBUG::2014-09-25 17:52:22,446::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:22,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:22,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134 s, 76.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:52:24,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:24,566::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205288 s, 49.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:24,836::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:24,837::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:24,837::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:24,837::__init__::232::IOProcess::(_processLogs) (109695) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:24,838::__init__::232::IOProcess::(_processLogs) (109695) Queuing response >Thread-81::DEBUG::2014-09-25 17:52:24,839::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:52:24,856::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0056581 s, 65.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:24,857::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:24,857::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:24,858::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:24,858::__init__::232::IOProcess::(_processLogs) (109696) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:24,859::__init__::232::IOProcess::(_processLogs) (109696) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:25,461::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:25,463::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71300::DEBUG::2014-09-25 17:52:25,465::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:26,585::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:26,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013357 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:27,024::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:27,024::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:27,025::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:27,025::__init__::232::IOProcess::(_processLogs) (109697) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:27,025::__init__::232::IOProcess::(_processLogs) (109697) Queuing response >Thread-80::DEBUG::2014-09-25 17:52:27,027::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:52:27,035::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000600991 s, 629 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:27,037::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:27,037::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:27,037::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:27,037::__init__::232::IOProcess::(_processLogs) (109698) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:27,038::__init__::232::IOProcess::(_processLogs) (109698) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:28,481::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:28,482::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71301::DEBUG::2014-09-25 17:52:28,485::task::595::Storage.TaskManager.Task::(_updateState) Task=`e11b1c70-16ed-4bca-bf2f-73268a7def4c`::moving from state init -> state preparing >Thread-71301::INFO::2014-09-25 17:52:28,485::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71301::INFO::2014-09-25 17:52:28,486::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000534334', 'lastCheck': '7.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000600991', 'lastCheck': '1.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0056581', 'lastCheck': '3.6', 'valid': True}} >Thread-71301::DEBUG::2014-09-25 17:52:28,486::task::1191::Storage.TaskManager.Task::(prepare) Task=`e11b1c70-16ed-4bca-bf2f-73268a7def4c`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000534334', 'lastCheck': '7.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000600991', 'lastCheck': '1.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0056581', 'lastCheck': '3.6', 'valid': True}} >Thread-71301::DEBUG::2014-09-25 17:52:28,486::task::595::Storage.TaskManager.Task::(_updateState) Task=`e11b1c70-16ed-4bca-bf2f-73268a7def4c`::moving from state preparing -> state finished >Thread-71301::DEBUG::2014-09-25 17:52:28,486::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71301::DEBUG::2014-09-25 17:52:28,487::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71301::DEBUG::2014-09-25 17:52:28,487::task::993::Storage.TaskManager.Task::(_decref) Task=`e11b1c70-16ed-4bca-bf2f-73268a7def4c`::ref 0 aborting False >Thread-71301::DEBUG::2014-09-25 17:52:28,489::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:28,512::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:28,513::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71302::DEBUG::2014-09-25 17:52:28,515::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:28,627::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:28,650::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134086 s, 76.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:30,550::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:30,550::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:30,550::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:30,550::__init__::232::IOProcess::(_processLogs) (109699) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:30,551::__init__::232::IOProcess::(_processLogs) (109699) Queuing response >Thread-79::DEBUG::2014-09-25 17:52:30,552::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:52:30,561::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000443001 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:30,562::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:30,562::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:30,563::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:30,563::__init__::232::IOProcess::(_processLogs) (109700) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:30,564::__init__::232::IOProcess::(_processLogs) (109700) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:30,565::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:30,565::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:30,565::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:30,566::__init__::232::IOProcess::(_processLogs) (109701) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:30,566::__init__::232::IOProcess::(_processLogs) (109701) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:30,567::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:30,567::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:30,567::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:30,568::__init__::232::IOProcess::(_processLogs) (109702) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:30,568::__init__::232::IOProcess::(_processLogs) (109702) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:30,569::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:30,569::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:30,569::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:30,570::__init__::232::IOProcess::(_processLogs) (109703) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:30,570::__init__::232::IOProcess::(_processLogs) (109703) Queuing response >Dummy-148::DEBUG::2014-09-25 17:52:30,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:30,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.01368 s, 74.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:31,573::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:31,575::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71303::DEBUG::2014-09-25 17:52:31,577::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:31,783::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:31,785::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71304::DEBUG::2014-09-25 17:52:31,785::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71304::DEBUG::2014-09-25 17:52:31,787::task::595::Storage.TaskManager.Task::(_updateState) Task=`56c6a3e4-4371-4910-814d-6f6543996520`::moving from state init -> state preparing >Thread-71304::INFO::2014-09-25 17:52:31,788::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71304::INFO::2014-09-25 17:52:31,788::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71304::DEBUG::2014-09-25 17:52:31,789::task::1191::Storage.TaskManager.Task::(prepare) Task=`56c6a3e4-4371-4910-814d-6f6543996520`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71304::DEBUG::2014-09-25 17:52:31,789::task::595::Storage.TaskManager.Task::(_updateState) Task=`56c6a3e4-4371-4910-814d-6f6543996520`::moving from state preparing -> state finished >Thread-71304::DEBUG::2014-09-25 17:52:31,789::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71304::DEBUG::2014-09-25 17:52:31,789::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71304::DEBUG::2014-09-25 17:52:31,789::task::993::Storage.TaskManager.Task::(_decref) Task=`56c6a3e4-4371-4910-814d-6f6543996520`::ref 0 aborting False >Thread-71304::DEBUG::2014-09-25 17:52:31,790::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71304::DEBUG::2014-09-25 17:52:31,790::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:31,807::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:31,808::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71305::DEBUG::2014-09-25 17:52:31,808::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71305::DEBUG::2014-09-25 17:52:31,810::task::595::Storage.TaskManager.Task::(_updateState) Task=`5dfccfdc-a53f-4514-9400-5c44cef5bae6`::moving from state init -> state preparing >Thread-71305::INFO::2014-09-25 17:52:31,810::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71305::DEBUG::2014-09-25 17:52:31,811::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7e19e7f0-64ec-4c58-858a-156893c066f0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71305::DEBUG::2014-09-25 17:52:31,811::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71305::DEBUG::2014-09-25 17:52:31,812::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71305::DEBUG::2014-09-25 17:52:31,812::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7e19e7f0-64ec-4c58-858a-156893c066f0`::Granted request >Thread-71305::DEBUG::2014-09-25 17:52:31,812::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5dfccfdc-a53f-4514-9400-5c44cef5bae6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71305::DEBUG::2014-09-25 17:52:31,813::task::993::Storage.TaskManager.Task::(_decref) Task=`5dfccfdc-a53f-4514-9400-5c44cef5bae6`::ref 1 aborting False >Thread-71305::INFO::2014-09-25 17:52:31,817::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71305::DEBUG::2014-09-25 17:52:31,817::task::1191::Storage.TaskManager.Task::(prepare) Task=`5dfccfdc-a53f-4514-9400-5c44cef5bae6`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71305::DEBUG::2014-09-25 17:52:31,818::task::595::Storage.TaskManager.Task::(_updateState) Task=`5dfccfdc-a53f-4514-9400-5c44cef5bae6`::moving from state preparing -> state finished >Thread-71305::DEBUG::2014-09-25 17:52:31,818::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71305::DEBUG::2014-09-25 17:52:31,818::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71305::DEBUG::2014-09-25 17:52:31,818::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71305::DEBUG::2014-09-25 17:52:31,819::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71305::DEBUG::2014-09-25 17:52:31,819::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71305::DEBUG::2014-09-25 17:52:31,819::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71305::DEBUG::2014-09-25 17:52:31,819::task::993::Storage.TaskManager.Task::(_decref) Task=`5dfccfdc-a53f-4514-9400-5c44cef5bae6`::ref 0 aborting False >Thread-71305::DEBUG::2014-09-25 17:52:31,820::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71305::DEBUG::2014-09-25 17:52:31,820::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:32,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:32,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013796 s, 74.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:34,592::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:34,594::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71306::DEBUG::2014-09-25 17:52:34,596::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:34,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:34,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132967 s, 77.0 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 17:52:34,861::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:52:34,864::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:34,866::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:34,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:34,866::__init__::232::IOProcess::(_processLogs) (109704) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:52:34,867::__init__::232::IOProcess::(_processLogs) (109704) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:34,869::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:34,871::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:34,873::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:34,874::__init__::232::IOProcess::(_processLogs) (109705) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:52:34,875::__init__::232::IOProcess::(_processLogs) (109705) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:34,877::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:34,879::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:34,880::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:34,881::__init__::232::IOProcess::(_processLogs) (109706) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:52:34,884::__init__::232::IOProcess::(_processLogs) (109706) Queuing response >Thread-81::DEBUG::2014-09-25 17:52:34,884::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:52:34,886::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:34,886::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:34,886::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:34,887::__init__::232::IOProcess::(_processLogs) (109707) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:52:34,913::__init__::232::IOProcess::(_processLogs) (109707) Queuing response >Thread-81::DEBUG::2014-09-25 17:52:34,913::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:52:34,915::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:34,915::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:34,915::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:34,916::__init__::232::IOProcess::(_processLogs) (109708) Got request for method 'readfile' >Thread-81::DEBUG::2014-09-25 17:52:34,916::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:52:34,916::__init__::232::IOProcess::(_processLogs) (109708) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:34,918::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:34,919::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:34,919::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:34,919::__init__::232::IOProcess::(_processLogs) (109709) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:34,919::__init__::232::IOProcess::(_processLogs) (109709) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:34,920::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:34,921::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:34,921::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:34,921::__init__::232::IOProcess::(_processLogs) (109710) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:52:34,922::__init__::232::IOProcess::(_processLogs) (109710) Queuing response >Thread-81::DEBUG::2014-09-25 17:52:34,922::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 17:52:34,923::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:52:34,923::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:52:34,924::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:34,924::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:34,924::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:34,924::__init__::232::IOProcess::(_processLogs) (109711) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:34,925::__init__::232::IOProcess::(_processLogs) (109711) Queuing response >Thread-81::DEBUG::2014-09-25 17:52:34,926::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:52:34,934::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00037561 s, 988 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:34,936::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:34,936::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:34,936::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:34,937::__init__::232::IOProcess::(_processLogs) (109712) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:34,938::__init__::232::IOProcess::(_processLogs) (109712) Queuing response >Dummy-148::DEBUG::2014-09-25 17:52:36,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:36,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158828 s, 64.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:37,039::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:37,040::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:37,040::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:37,040::__init__::232::IOProcess::(_processLogs) (109713) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:37,041::__init__::232::IOProcess::(_processLogs) (109713) Queuing response >Thread-80::DEBUG::2014-09-25 17:52:37,042::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:52:37,051::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00042562 s, 888 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:37,052::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:37,052::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:37,052::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:37,053::__init__::232::IOProcess::(_processLogs) (109714) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:37,053::__init__::232::IOProcess::(_processLogs) (109714) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:37,611::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:37,612::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71310::DEBUG::2014-09-25 17:52:37,614::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:38,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:38,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136679 s, 74.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:40,571::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:40,571::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:40,572::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:40,572::__init__::232::IOProcess::(_processLogs) (109715) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:40,572::__init__::232::IOProcess::(_processLogs) (109715) Queuing response >Thread-79::DEBUG::2014-09-25 17:52:40,574::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:52:40,582::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000436249 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:40,583::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:40,584::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:40,584::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:40,584::__init__::232::IOProcess::(_processLogs) (109716) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:40,585::__init__::232::IOProcess::(_processLogs) (109716) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:40,586::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:40,587::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:40,588::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:40,588::__init__::232::IOProcess::(_processLogs) (109717) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:40,588::__init__::232::IOProcess::(_processLogs) (109717) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:40,588::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:40,589::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:40,590::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:40,590::__init__::232::IOProcess::(_processLogs) (109718) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:40,590::__init__::232::IOProcess::(_processLogs) (109718) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:40,590::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:40,591::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:40,591::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:40,592::__init__::232::IOProcess::(_processLogs) (109719) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:40,592::__init__::232::IOProcess::(_processLogs) (109719) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:40,630::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:40,632::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71311::DEBUG::2014-09-25 17:52:40,634::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:40,887::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:40,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133905 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:41,849::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71312::DEBUG::2014-09-25 17:52:41,850::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:52:41,850::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71312::DEBUG::2014-09-25 17:52:41,853::task::595::Storage.TaskManager.Task::(_updateState) Task=`9e75ec87-d239-483c-9469-ee6d8078bf5d`::moving from state init -> state preparing >Thread-71312::INFO::2014-09-25 17:52:41,853::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71312::INFO::2014-09-25 17:52:41,854::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71312::DEBUG::2014-09-25 17:52:41,854::task::1191::Storage.TaskManager.Task::(prepare) Task=`9e75ec87-d239-483c-9469-ee6d8078bf5d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71312::DEBUG::2014-09-25 17:52:41,854::task::595::Storage.TaskManager.Task::(_updateState) Task=`9e75ec87-d239-483c-9469-ee6d8078bf5d`::moving from state preparing -> state finished >Thread-71312::DEBUG::2014-09-25 17:52:41,855::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71312::DEBUG::2014-09-25 17:52:41,855::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71312::DEBUG::2014-09-25 17:52:41,855::task::993::Storage.TaskManager.Task::(_decref) Task=`9e75ec87-d239-483c-9469-ee6d8078bf5d`::ref 0 aborting False >Thread-71312::DEBUG::2014-09-25 17:52:41,856::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71312::DEBUG::2014-09-25 17:52:41,857::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:41,872::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:41,873::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71313::DEBUG::2014-09-25 17:52:41,873::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71313::DEBUG::2014-09-25 17:52:41,875::task::595::Storage.TaskManager.Task::(_updateState) Task=`ae2d8f1c-a0c2-4bfb-bbf3-eafb7ee71069`::moving from state init -> state preparing >Thread-71313::INFO::2014-09-25 17:52:41,875::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71313::DEBUG::2014-09-25 17:52:41,876::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`070cf6d9-252b-4978-8116-74b125c68b24`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71313::DEBUG::2014-09-25 17:52:41,876::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71313::DEBUG::2014-09-25 17:52:41,876::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71313::DEBUG::2014-09-25 17:52:41,877::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`070cf6d9-252b-4978-8116-74b125c68b24`::Granted request >Thread-71313::DEBUG::2014-09-25 17:52:41,878::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ae2d8f1c-a0c2-4bfb-bbf3-eafb7ee71069`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71313::DEBUG::2014-09-25 17:52:41,879::task::993::Storage.TaskManager.Task::(_decref) Task=`ae2d8f1c-a0c2-4bfb-bbf3-eafb7ee71069`::ref 1 aborting False >Thread-71313::INFO::2014-09-25 17:52:41,883::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71313::DEBUG::2014-09-25 17:52:41,883::task::1191::Storage.TaskManager.Task::(prepare) Task=`ae2d8f1c-a0c2-4bfb-bbf3-eafb7ee71069`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71313::DEBUG::2014-09-25 17:52:41,884::task::595::Storage.TaskManager.Task::(_updateState) Task=`ae2d8f1c-a0c2-4bfb-bbf3-eafb7ee71069`::moving from state preparing -> state finished >Thread-71313::DEBUG::2014-09-25 17:52:41,884::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71313::DEBUG::2014-09-25 17:52:41,884::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71313::DEBUG::2014-09-25 17:52:41,884::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71313::DEBUG::2014-09-25 17:52:41,885::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71313::DEBUG::2014-09-25 17:52:41,885::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71313::DEBUG::2014-09-25 17:52:41,885::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71313::DEBUG::2014-09-25 17:52:41,886::task::993::Storage.TaskManager.Task::(_decref) Task=`ae2d8f1c-a0c2-4bfb-bbf3-eafb7ee71069`::ref 0 aborting False >Thread-71313::DEBUG::2014-09-25 17:52:41,886::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71313::DEBUG::2014-09-25 17:52:41,887::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:42,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:42,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136334 s, 75.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:43,649::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:43,651::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71314::DEBUG::2014-09-25 17:52:43,653::task::595::Storage.TaskManager.Task::(_updateState) Task=`b3312722-b0d8-484e-b189-6a975b61ae9e`::moving from state init -> state preparing >Thread-71314::INFO::2014-09-25 17:52:43,654::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71314::INFO::2014-09-25 17:52:43,654::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000436249', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00042562', 'lastCheck': '6.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00037561', 'lastCheck': '8.7', 'valid': True}} >Thread-71314::DEBUG::2014-09-25 17:52:43,655::task::1191::Storage.TaskManager.Task::(prepare) Task=`b3312722-b0d8-484e-b189-6a975b61ae9e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000436249', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00042562', 'lastCheck': '6.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00037561', 'lastCheck': '8.7', 'valid': True}} >Thread-71314::DEBUG::2014-09-25 17:52:43,655::task::595::Storage.TaskManager.Task::(_updateState) Task=`b3312722-b0d8-484e-b189-6a975b61ae9e`::moving from state preparing -> state finished >Thread-71314::DEBUG::2014-09-25 17:52:43,655::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71314::DEBUG::2014-09-25 17:52:43,655::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71314::DEBUG::2014-09-25 17:52:43,655::task::993::Storage.TaskManager.Task::(_decref) Task=`b3312722-b0d8-484e-b189-6a975b61ae9e`::ref 0 aborting False >Thread-71314::DEBUG::2014-09-25 17:52:43,664::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:43,686::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:43,687::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71315::DEBUG::2014-09-25 17:52:43,689::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:52:44,939::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:44,939::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:44,940::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:52:44,941::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:52:44,941::__init__::232::IOProcess::(_processLogs) (109720) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:44,950::__init__::232::IOProcess::(_processLogs) (109720) Queuing response >Thread-81::DEBUG::2014-09-25 17:52:44,951::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000384314 s, 965 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:44,952::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:44,952::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:44,953::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:44,954::__init__::232::IOProcess::(_processLogs) (109721) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:44,954::__init__::232::IOProcess::(_processLogs) (109721) Queuing response >Dummy-148::DEBUG::2014-09-25 17:52:44,971::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:44,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013377 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:46,747::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:46,750::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71316::DEBUG::2014-09-25 17:52:46,756::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:47,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:47,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0121182 s, 84.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:47,055::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:47,055::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:47,055::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:47,056::__init__::232::IOProcess::(_processLogs) (109722) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:47,056::__init__::232::IOProcess::(_processLogs) (109722) Queuing response >Thread-80::DEBUG::2014-09-25 17:52:47,057::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:52:47,064::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000405505 s, 932 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:47,065::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:47,065::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:47,066::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:47,066::__init__::232::IOProcess::(_processLogs) (109723) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:47,066::__init__::232::IOProcess::(_processLogs) (109723) Queuing response >Dummy-148::DEBUG::2014-09-25 17:52:49,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:49,067::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133354 s, 76.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:49,774::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:49,775::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71317::DEBUG::2014-09-25 17:52:49,777::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:52:50,593::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:50,593::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:50,594::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:50,594::__init__::232::IOProcess::(_processLogs) (109724) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:50,594::__init__::232::IOProcess::(_processLogs) (109724) Queuing response >Thread-79::DEBUG::2014-09-25 17:52:50,595::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:52:50,603::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000329639 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:50,605::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:50,605::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:50,605::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:50,607::__init__::232::IOProcess::(_processLogs) (109725) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:50,607::__init__::232::IOProcess::(_processLogs) (109725) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:50,607::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:50,608::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:50,608::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:50,609::__init__::232::IOProcess::(_processLogs) (109726) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:50,609::__init__::232::IOProcess::(_processLogs) (109726) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:50,610::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:50,610::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:50,610::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:50,611::__init__::232::IOProcess::(_processLogs) (109727) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:50,611::__init__::232::IOProcess::(_processLogs) (109727) Queuing response >Thread-29::DEBUG::2014-09-25 17:52:50,612::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:50,612::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:50,612::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:50,613::__init__::232::IOProcess::(_processLogs) (109728) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:52:50,614::__init__::232::IOProcess::(_processLogs) (109728) Queuing response >Dummy-148::DEBUG::2014-09-25 17:52:51,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:51,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135142 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:51,914::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:51,916::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71318::DEBUG::2014-09-25 17:52:51,917::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71318::DEBUG::2014-09-25 17:52:51,920::task::595::Storage.TaskManager.Task::(_updateState) Task=`9168ee0c-18ce-4378-98ee-d2b4e409a631`::moving from state init -> state preparing >Thread-71318::INFO::2014-09-25 17:52:51,920::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71318::INFO::2014-09-25 17:52:51,921::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71318::DEBUG::2014-09-25 17:52:51,921::task::1191::Storage.TaskManager.Task::(prepare) Task=`9168ee0c-18ce-4378-98ee-d2b4e409a631`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71318::DEBUG::2014-09-25 17:52:51,921::task::595::Storage.TaskManager.Task::(_updateState) Task=`9168ee0c-18ce-4378-98ee-d2b4e409a631`::moving from state preparing -> state finished >Thread-71318::DEBUG::2014-09-25 17:52:51,922::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71318::DEBUG::2014-09-25 17:52:51,922::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71318::DEBUG::2014-09-25 17:52:51,922::task::993::Storage.TaskManager.Task::(_decref) Task=`9168ee0c-18ce-4378-98ee-d2b4e409a631`::ref 0 aborting False >Thread-71318::DEBUG::2014-09-25 17:52:51,922::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71318::DEBUG::2014-09-25 17:52:51,923::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:51,938::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71319::DEBUG::2014-09-25 17:52:51,939::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:52:51,939::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71319::DEBUG::2014-09-25 17:52:51,941::task::595::Storage.TaskManager.Task::(_updateState) Task=`63bda177-ff37-409b-b7e5-313e455a9a08`::moving from state init -> state preparing >Thread-71319::INFO::2014-09-25 17:52:51,941::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71319::DEBUG::2014-09-25 17:52:51,942::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cf37f4cc-12e2-4458-b667-9eb15cf664ca`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71319::DEBUG::2014-09-25 17:52:51,942::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71319::DEBUG::2014-09-25 17:52:51,943::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71319::DEBUG::2014-09-25 17:52:51,943::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cf37f4cc-12e2-4458-b667-9eb15cf664ca`::Granted request >Thread-71319::DEBUG::2014-09-25 17:52:51,943::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`63bda177-ff37-409b-b7e5-313e455a9a08`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71319::DEBUG::2014-09-25 17:52:51,944::task::993::Storage.TaskManager.Task::(_decref) Task=`63bda177-ff37-409b-b7e5-313e455a9a08`::ref 1 aborting False >Thread-71319::INFO::2014-09-25 17:52:51,948::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71319::DEBUG::2014-09-25 17:52:51,948::task::1191::Storage.TaskManager.Task::(prepare) Task=`63bda177-ff37-409b-b7e5-313e455a9a08`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71319::DEBUG::2014-09-25 17:52:51,949::task::595::Storage.TaskManager.Task::(_updateState) Task=`63bda177-ff37-409b-b7e5-313e455a9a08`::moving from state preparing -> state finished >Thread-71319::DEBUG::2014-09-25 17:52:51,949::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71319::DEBUG::2014-09-25 17:52:51,949::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71319::DEBUG::2014-09-25 17:52:51,949::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71319::DEBUG::2014-09-25 17:52:51,950::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71319::DEBUG::2014-09-25 17:52:51,950::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71319::DEBUG::2014-09-25 17:52:51,950::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71319::DEBUG::2014-09-25 17:52:51,950::task::993::Storage.TaskManager.Task::(_decref) Task=`63bda177-ff37-409b-b7e5-313e455a9a08`::ref 0 aborting False >Thread-71319::DEBUG::2014-09-25 17:52:51,951::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71319::DEBUG::2014-09-25 17:52:51,951::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:52,794::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:52,796::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71320::DEBUG::2014-09-25 17:52:52,797::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:53,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:53,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135733 s, 75.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:54,955::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:54,955::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:54,955::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:54,956::__init__::232::IOProcess::(_processLogs) (109729) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:54,956::__init__::232::IOProcess::(_processLogs) (109729) Queuing response >Thread-81::DEBUG::2014-09-25 17:52:54,957::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:52:54,966::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000491414 s, 755 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:54,967::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:54,968::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:54,968::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:54,968::__init__::232::IOProcess::(_processLogs) (109730) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:54,969::__init__::232::IOProcess::(_processLogs) (109730) Queuing response >Dummy-148::DEBUG::2014-09-25 17:52:55,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:55,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137445 s, 74.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:55,812::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:55,813::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71321::DEBUG::2014-09-25 17:52:55,815::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:52:57,068::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:57,068::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:57,069::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:57,069::__init__::232::IOProcess::(_processLogs) (109731) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:57,069::__init__::232::IOProcess::(_processLogs) (109731) Queuing response >Thread-80::DEBUG::2014-09-25 17:52:57,071::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:52:57,079::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000484279 s, 781 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:52:57,080::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:52:57,080::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:52:57,081::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:52:57,081::__init__::232::IOProcess::(_processLogs) (109732) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:52:57,082::__init__::232::IOProcess::(_processLogs) (109732) Queuing response >Dummy-148::DEBUG::2014-09-25 17:52:57,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:57,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.01386 s, 73.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:58,829::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:58,830::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71322::DEBUG::2014-09-25 17:52:58,833::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f91acf6-201a-43bf-93f7-48a20f78e0b2`::moving from state init -> state preparing >Thread-71322::INFO::2014-09-25 17:52:58,833::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71322::INFO::2014-09-25 17:52:58,833::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000329639', 'lastCheck': '8.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000484279', 'lastCheck': '1.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000491414', 'lastCheck': '3.9', 'valid': True}} >Thread-71322::DEBUG::2014-09-25 17:52:58,834::task::1191::Storage.TaskManager.Task::(prepare) Task=`9f91acf6-201a-43bf-93f7-48a20f78e0b2`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000329639', 'lastCheck': '8.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000484279', 'lastCheck': '1.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000491414', 'lastCheck': '3.9', 'valid': True}} >Thread-71322::DEBUG::2014-09-25 17:52:58,834::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f91acf6-201a-43bf-93f7-48a20f78e0b2`::moving from state preparing -> state finished >Thread-71322::DEBUG::2014-09-25 17:52:58,834::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71322::DEBUG::2014-09-25 17:52:58,834::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71322::DEBUG::2014-09-25 17:52:58,835::task::993::Storage.TaskManager.Task::(_decref) Task=`9f91acf6-201a-43bf-93f7-48a20f78e0b2`::ref 0 aborting False >Thread-71322::DEBUG::2014-09-25 17:52:58,837::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:52:58,860::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:52:58,861::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71323::DEBUG::2014-09-25 17:52:58,863::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:52:59,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:52:59,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151125 s, 67.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:00,614::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:00,614::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:00,615::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:53:00,616::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:53:00,617::__init__::232::IOProcess::(_processLogs) (109733) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:00,625::__init__::232::IOProcess::(_processLogs) (109733) Queuing response >Thread-79::DEBUG::2014-09-25 17:53:00,626::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000461417 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:00,627::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:00,627::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:00,628::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:00,628::__init__::232::IOProcess::(_processLogs) (109734) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:00,629::__init__::232::IOProcess::(_processLogs) (109734) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:00,630::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:00,631::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:00,631::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:00,631::__init__::232::IOProcess::(_processLogs) (109735) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:00,632::__init__::232::IOProcess::(_processLogs) (109735) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:00,632::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:00,632::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:00,633::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:00,633::__init__::232::IOProcess::(_processLogs) (109736) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:00,634::__init__::232::IOProcess::(_processLogs) (109736) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:00,634::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:00,634::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:00,635::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:00,636::__init__::232::IOProcess::(_processLogs) (109737) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:00,636::__init__::232::IOProcess::(_processLogs) (109737) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:01,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:01,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135705 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:01,925::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:01,926::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71324::DEBUG::2014-09-25 17:53:01,928::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:01,978::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71325::DEBUG::2014-09-25 17:53:01,980::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:53:01,980::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71325::DEBUG::2014-09-25 17:53:01,983::task::595::Storage.TaskManager.Task::(_updateState) Task=`127d7894-2ebd-4399-a83e-6e30b2d5154e`::moving from state init -> state preparing >Thread-71325::INFO::2014-09-25 17:53:01,983::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71325::INFO::2014-09-25 17:53:01,983::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71325::DEBUG::2014-09-25 17:53:01,984::task::1191::Storage.TaskManager.Task::(prepare) Task=`127d7894-2ebd-4399-a83e-6e30b2d5154e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71325::DEBUG::2014-09-25 17:53:01,984::task::595::Storage.TaskManager.Task::(_updateState) Task=`127d7894-2ebd-4399-a83e-6e30b2d5154e`::moving from state preparing -> state finished >Thread-71325::DEBUG::2014-09-25 17:53:01,984::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71325::DEBUG::2014-09-25 17:53:01,984::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71325::DEBUG::2014-09-25 17:53:01,985::task::993::Storage.TaskManager.Task::(_decref) Task=`127d7894-2ebd-4399-a83e-6e30b2d5154e`::ref 0 aborting False >Thread-71325::DEBUG::2014-09-25 17:53:01,985::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71325::DEBUG::2014-09-25 17:53:01,985::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:01,998::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71326::DEBUG::2014-09-25 17:53:01,999::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:53:01,999::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71326::DEBUG::2014-09-25 17:53:02,001::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b2c7595-b92e-4b8c-bb02-6acfd3aa313c`::moving from state init -> state preparing >Thread-71326::INFO::2014-09-25 17:53:02,002::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71326::DEBUG::2014-09-25 17:53:02,002::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7e9613c4-4f5e-4421-a5eb-a0fbed3d7b58`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71326::DEBUG::2014-09-25 17:53:02,003::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71326::DEBUG::2014-09-25 17:53:02,003::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71326::DEBUG::2014-09-25 17:53:02,003::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7e9613c4-4f5e-4421-a5eb-a0fbed3d7b58`::Granted request >Thread-71326::DEBUG::2014-09-25 17:53:02,004::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4b2c7595-b92e-4b8c-bb02-6acfd3aa313c`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71326::DEBUG::2014-09-25 17:53:02,004::task::993::Storage.TaskManager.Task::(_decref) Task=`4b2c7595-b92e-4b8c-bb02-6acfd3aa313c`::ref 1 aborting False >Thread-71326::INFO::2014-09-25 17:53:02,009::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71326::DEBUG::2014-09-25 17:53:02,009::task::1191::Storage.TaskManager.Task::(prepare) Task=`4b2c7595-b92e-4b8c-bb02-6acfd3aa313c`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71326::DEBUG::2014-09-25 17:53:02,009::task::595::Storage.TaskManager.Task::(_updateState) Task=`4b2c7595-b92e-4b8c-bb02-6acfd3aa313c`::moving from state preparing -> state finished >Thread-71326::DEBUG::2014-09-25 17:53:02,009::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71326::DEBUG::2014-09-25 17:53:02,010::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71326::DEBUG::2014-09-25 17:53:02,010::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71326::DEBUG::2014-09-25 17:53:02,010::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71326::DEBUG::2014-09-25 17:53:02,011::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71326::DEBUG::2014-09-25 17:53:02,011::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71326::DEBUG::2014-09-25 17:53:02,011::task::993::Storage.TaskManager.Task::(_decref) Task=`4b2c7595-b92e-4b8c-bb02-6acfd3aa313c`::ref 0 aborting False >Thread-71326::DEBUG::2014-09-25 17:53:02,012::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7251951616', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71326::DEBUG::2014-09-25 17:53:02,012::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:03,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:03,370::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135594 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:04,942::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:04,943::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71327::DEBUG::2014-09-25 17:53:04,945::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:53:04,971::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:04,971::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:04,972::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:04,972::__init__::232::IOProcess::(_processLogs) (109738) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:04,972::__init__::232::IOProcess::(_processLogs) (109738) Queuing response >Thread-81::DEBUG::2014-09-25 17:53:04,973::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:53:04,982::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000347121 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:04,983::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:04,983::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:04,984::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:04,984::__init__::232::IOProcess::(_processLogs) (109739) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:04,985::__init__::232::IOProcess::(_processLogs) (109739) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:05,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:05,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135994 s, 75.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:07,084::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:07,084::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:07,084::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:07,085::__init__::232::IOProcess::(_processLogs) (109740) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:07,085::__init__::232::IOProcess::(_processLogs) (109740) Queuing response >Thread-80::DEBUG::2014-09-25 17:53:07,086::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:53:07,096::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000618385 s, 611 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:07,097::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:07,097::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:07,097::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:07,098::__init__::232::IOProcess::(_processLogs) (109741) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:07,098::__init__::232::IOProcess::(_processLogs) (109741) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:07,432::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:07,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134707 s, 76.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:07,960::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:07,961::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71328::DEBUG::2014-09-25 17:53:07,963::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:09,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:09,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134555 s, 76.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:10,636::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:10,637::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:10,637::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:10,637::__init__::232::IOProcess::(_processLogs) (109742) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:10,638::__init__::232::IOProcess::(_processLogs) (109742) Queuing response >Thread-79::DEBUG::2014-09-25 17:53:10,639::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:53:10,648::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000409324 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:10,650::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:10,650::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:10,650::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:10,650::__init__::232::IOProcess::(_processLogs) (109743) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:10,651::__init__::232::IOProcess::(_processLogs) (109743) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:10,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:10,653::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:10,654::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:10,654::__init__::232::IOProcess::(_processLogs) (109744) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:10,654::__init__::232::IOProcess::(_processLogs) (109744) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:10,655::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:10,655::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:10,656::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:10,656::__init__::232::IOProcess::(_processLogs) (109745) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:10,656::__init__::232::IOProcess::(_processLogs) (109745) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:10,657::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:10,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:10,659::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:10,659::__init__::232::IOProcess::(_processLogs) (109746) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:10,659::__init__::232::IOProcess::(_processLogs) (109746) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:10,976::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:10,978::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71329::DEBUG::2014-09-25 17:53:10,980::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:11,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:11,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164806 s, 62.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:12,039::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71330::DEBUG::2014-09-25 17:53:12,041::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:53:12,041::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71330::DEBUG::2014-09-25 17:53:12,044::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2b0f8d0-6b63-4d50-9264-ef8f93c69517`::moving from state init -> state preparing >Thread-71330::INFO::2014-09-25 17:53:12,044::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71330::INFO::2014-09-25 17:53:12,045::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71330::DEBUG::2014-09-25 17:53:12,045::task::1191::Storage.TaskManager.Task::(prepare) Task=`c2b0f8d0-6b63-4d50-9264-ef8f93c69517`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71330::DEBUG::2014-09-25 17:53:12,045::task::595::Storage.TaskManager.Task::(_updateState) Task=`c2b0f8d0-6b63-4d50-9264-ef8f93c69517`::moving from state preparing -> state finished >Thread-71330::DEBUG::2014-09-25 17:53:12,045::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71330::DEBUG::2014-09-25 17:53:12,045::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71330::DEBUG::2014-09-25 17:53:12,046::task::993::Storage.TaskManager.Task::(_decref) Task=`c2b0f8d0-6b63-4d50-9264-ef8f93c69517`::ref 0 aborting False >Thread-71330::DEBUG::2014-09-25 17:53:12,046::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71330::DEBUG::2014-09-25 17:53:12,046::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:12,060::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:12,062::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71331::DEBUG::2014-09-25 17:53:12,062::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71331::DEBUG::2014-09-25 17:53:12,064::task::595::Storage.TaskManager.Task::(_updateState) Task=`0aad8886-7249-4709-a809-820a9d5ad794`::moving from state init -> state preparing >Thread-71331::INFO::2014-09-25 17:53:12,064::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71331::DEBUG::2014-09-25 17:53:12,065::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f52168f2-f956-46cf-9097-14fcb05e7dfa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71331::DEBUG::2014-09-25 17:53:12,065::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71331::DEBUG::2014-09-25 17:53:12,065::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71331::DEBUG::2014-09-25 17:53:12,066::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f52168f2-f956-46cf-9097-14fcb05e7dfa`::Granted request >Thread-71331::DEBUG::2014-09-25 17:53:12,066::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0aad8886-7249-4709-a809-820a9d5ad794`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71331::DEBUG::2014-09-25 17:53:12,066::task::993::Storage.TaskManager.Task::(_decref) Task=`0aad8886-7249-4709-a809-820a9d5ad794`::ref 1 aborting False >Thread-71331::INFO::2014-09-25 17:53:12,071::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71331::DEBUG::2014-09-25 17:53:12,071::task::1191::Storage.TaskManager.Task::(prepare) Task=`0aad8886-7249-4709-a809-820a9d5ad794`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71331::DEBUG::2014-09-25 17:53:12,071::task::595::Storage.TaskManager.Task::(_updateState) Task=`0aad8886-7249-4709-a809-820a9d5ad794`::moving from state preparing -> state finished >Thread-71331::DEBUG::2014-09-25 17:53:12,072::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71331::DEBUG::2014-09-25 17:53:12,072::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71331::DEBUG::2014-09-25 17:53:12,072::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71331::DEBUG::2014-09-25 17:53:12,072::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71331::DEBUG::2014-09-25 17:53:12,073::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71331::DEBUG::2014-09-25 17:53:12,073::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71331::DEBUG::2014-09-25 17:53:12,073::task::993::Storage.TaskManager.Task::(_decref) Task=`0aad8886-7249-4709-a809-820a9d5ad794`::ref 0 aborting False >Thread-71331::DEBUG::2014-09-25 17:53:12,074::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71331::DEBUG::2014-09-25 17:53:12,074::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:13,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:13,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136541 s, 75.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:13,995::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:13,996::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71332::DEBUG::2014-09-25 17:53:13,999::task::595::Storage.TaskManager.Task::(_updateState) Task=`a88f9c54-d4bc-4926-8a35-931836d4839d`::moving from state init -> state preparing >Thread-71332::INFO::2014-09-25 17:53:13,999::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71332::INFO::2014-09-25 17:53:14,000::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409324', 'lastCheck': '3.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000618385', 'lastCheck': '6.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000347121', 'lastCheck': '9.0', 'valid': True}} >Thread-71332::DEBUG::2014-09-25 17:53:14,000::task::1191::Storage.TaskManager.Task::(prepare) Task=`a88f9c54-d4bc-4926-8a35-931836d4839d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409324', 'lastCheck': '3.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000618385', 'lastCheck': '6.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000347121', 'lastCheck': '9.0', 'valid': True}} >Thread-71332::DEBUG::2014-09-25 17:53:14,000::task::595::Storage.TaskManager.Task::(_updateState) Task=`a88f9c54-d4bc-4926-8a35-931836d4839d`::moving from state preparing -> state finished >Thread-71332::DEBUG::2014-09-25 17:53:14,000::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71332::DEBUG::2014-09-25 17:53:14,001::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71332::DEBUG::2014-09-25 17:53:14,001::task::993::Storage.TaskManager.Task::(_decref) Task=`a88f9c54-d4bc-4926-8a35-931836d4839d`::ref 0 aborting False >Thread-71332::DEBUG::2014-09-25 17:53:14,003::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:14,026::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:14,027::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71333::DEBUG::2014-09-25 17:53:14,029::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:53:14,986::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:14,987::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:14,987::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:14,987::__init__::232::IOProcess::(_processLogs) (109747) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:14,988::__init__::232::IOProcess::(_processLogs) (109747) Queuing response >Thread-81::DEBUG::2014-09-25 17:53:14,988::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:53:14,997::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000381682 s, 972 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:14,998::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:14,998::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:14,999::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:14,999::__init__::232::IOProcess::(_processLogs) (109748) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:14,999::__init__::232::IOProcess::(_processLogs) (109748) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:15,609::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:15,637::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143157 s, 71.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:17,089::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:17,090::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71334::DEBUG::2014-09-25 17:53:17,091::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:53:17,100::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:17,100::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:17,101::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:17,101::__init__::232::IOProcess::(_processLogs) (109749) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:17,101::__init__::232::IOProcess::(_processLogs) (109749) Queuing response >Thread-80::DEBUG::2014-09-25 17:53:17,102::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:53:17,108::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000445155 s, 849 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:17,109::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:17,109::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:17,109::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:17,110::__init__::232::IOProcess::(_processLogs) (109750) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:17,110::__init__::232::IOProcess::(_processLogs) (109750) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:17,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:17,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013534 s, 75.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:53:19,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:19,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013481 s, 76.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:20,106::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:20,107::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71335::DEBUG::2014-09-25 17:53:20,109::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:53:20,659::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:20,660::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:20,660::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:20,660::__init__::232::IOProcess::(_processLogs) (109751) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:20,660::__init__::232::IOProcess::(_processLogs) (109751) Queuing response >Thread-79::DEBUG::2014-09-25 17:53:20,661::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:53:20,670::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000443211 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:20,671::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:20,672::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:20,672::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:20,672::__init__::232::IOProcess::(_processLogs) (109752) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:20,673::__init__::232::IOProcess::(_processLogs) (109752) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:20,675::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:20,675::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:20,676::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:20,676::__init__::232::IOProcess::(_processLogs) (109753) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:20,676::__init__::232::IOProcess::(_processLogs) (109753) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:20,677::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:20,677::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:20,678::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:20,678::__init__::232::IOProcess::(_processLogs) (109754) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:20,678::__init__::232::IOProcess::(_processLogs) (109754) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:20,679::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:20,679::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:20,679::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:20,679::__init__::232::IOProcess::(_processLogs) (109755) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:20,680::__init__::232::IOProcess::(_processLogs) (109755) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:21,741::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:21,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146606 s, 69.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:22,103::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71336::DEBUG::2014-09-25 17:53:22,105::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:53:22,105::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71336::DEBUG::2014-09-25 17:53:22,107::task::595::Storage.TaskManager.Task::(_updateState) Task=`7abc68c4-71c4-4bda-a105-7936d33074d1`::moving from state init -> state preparing >Thread-71336::INFO::2014-09-25 17:53:22,107::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71336::INFO::2014-09-25 17:53:22,108::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71336::DEBUG::2014-09-25 17:53:22,108::task::1191::Storage.TaskManager.Task::(prepare) Task=`7abc68c4-71c4-4bda-a105-7936d33074d1`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71336::DEBUG::2014-09-25 17:53:22,109::task::595::Storage.TaskManager.Task::(_updateState) Task=`7abc68c4-71c4-4bda-a105-7936d33074d1`::moving from state preparing -> state finished >Thread-71336::DEBUG::2014-09-25 17:53:22,109::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71336::DEBUG::2014-09-25 17:53:22,109::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71336::DEBUG::2014-09-25 17:53:22,109::task::993::Storage.TaskManager.Task::(_decref) Task=`7abc68c4-71c4-4bda-a105-7936d33074d1`::ref 0 aborting False >Thread-71336::DEBUG::2014-09-25 17:53:22,110::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71336::DEBUG::2014-09-25 17:53:22,110::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:22,121::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:22,122::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71337::DEBUG::2014-09-25 17:53:22,122::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71337::DEBUG::2014-09-25 17:53:22,124::task::595::Storage.TaskManager.Task::(_updateState) Task=`1d72e1f2-b313-4751-83d3-eb1e66961103`::moving from state init -> state preparing >Thread-71337::INFO::2014-09-25 17:53:22,124::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71337::DEBUG::2014-09-25 17:53:22,125::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2bede55b-0315-4812-acce-0e024a9749bc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71337::DEBUG::2014-09-25 17:53:22,125::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71337::DEBUG::2014-09-25 17:53:22,126::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71337::DEBUG::2014-09-25 17:53:22,126::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2bede55b-0315-4812-acce-0e024a9749bc`::Granted request >Thread-71337::DEBUG::2014-09-25 17:53:22,126::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1d72e1f2-b313-4751-83d3-eb1e66961103`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71337::DEBUG::2014-09-25 17:53:22,127::task::993::Storage.TaskManager.Task::(_decref) Task=`1d72e1f2-b313-4751-83d3-eb1e66961103`::ref 1 aborting False >Thread-71337::INFO::2014-09-25 17:53:22,131::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71337::DEBUG::2014-09-25 17:53:22,131::task::1191::Storage.TaskManager.Task::(prepare) Task=`1d72e1f2-b313-4751-83d3-eb1e66961103`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71337::DEBUG::2014-09-25 17:53:22,131::task::595::Storage.TaskManager.Task::(_updateState) Task=`1d72e1f2-b313-4751-83d3-eb1e66961103`::moving from state preparing -> state finished >Thread-71337::DEBUG::2014-09-25 17:53:22,132::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71337::DEBUG::2014-09-25 17:53:22,132::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71337::DEBUG::2014-09-25 17:53:22,132::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71337::DEBUG::2014-09-25 17:53:22,133::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71337::DEBUG::2014-09-25 17:53:22,133::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71337::DEBUG::2014-09-25 17:53:22,133::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71337::DEBUG::2014-09-25 17:53:22,133::task::993::Storage.TaskManager.Task::(_decref) Task=`1d72e1f2-b313-4751-83d3-eb1e66961103`::ref 0 aborting False >Thread-71337::DEBUG::2014-09-25 17:53:22,134::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71337::DEBUG::2014-09-25 17:53:22,134::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:23,121::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:23,122::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71338::DEBUG::2014-09-25 17:53:23,124::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:23,786::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:23,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013258 s, 77.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:25,002::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:25,002::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:25,003::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:25,003::__init__::232::IOProcess::(_processLogs) (109756) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:25,003::__init__::232::IOProcess::(_processLogs) (109756) Queuing response >Thread-81::DEBUG::2014-09-25 17:53:25,005::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:53:25,014::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000354647 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:25,015::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:25,015::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:25,015::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:25,016::__init__::232::IOProcess::(_processLogs) (109757) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:25,016::__init__::232::IOProcess::(_processLogs) (109757) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:25,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:25,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0206084 s, 49.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:26,134::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:26,136::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71339::DEBUG::2014-09-25 17:53:26,137::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:53:27,112::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:27,112::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:27,113::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:27,113::__init__::232::IOProcess::(_processLogs) (109758) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:27,113::__init__::232::IOProcess::(_processLogs) (109758) Queuing response >Thread-80::DEBUG::2014-09-25 17:53:27,114::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:53:27,123::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000448294 s, 843 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:27,124::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:27,124::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:27,124::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:27,125::__init__::232::IOProcess::(_processLogs) (109759) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:27,125::__init__::232::IOProcess::(_processLogs) (109759) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:27,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:27,912::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135928 s, 75.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:29,148::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:29,149::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71340::DEBUG::2014-09-25 17:53:29,152::task::595::Storage.TaskManager.Task::(_updateState) Task=`34560840-bbbe-4789-9d7c-b4e7d8b16b31`::moving from state init -> state preparing >Thread-71340::INFO::2014-09-25 17:53:29,152::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71340::INFO::2014-09-25 17:53:29,153::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443211', 'lastCheck': '8.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000448294', 'lastCheck': '2.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000354647', 'lastCheck': '4.1', 'valid': True}} >Thread-71340::DEBUG::2014-09-25 17:53:29,153::task::1191::Storage.TaskManager.Task::(prepare) Task=`34560840-bbbe-4789-9d7c-b4e7d8b16b31`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000443211', 'lastCheck': '8.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000448294', 'lastCheck': '2.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000354647', 'lastCheck': '4.1', 'valid': True}} >Thread-71340::DEBUG::2014-09-25 17:53:29,153::task::595::Storage.TaskManager.Task::(_updateState) Task=`34560840-bbbe-4789-9d7c-b4e7d8b16b31`::moving from state preparing -> state finished >Thread-71340::DEBUG::2014-09-25 17:53:29,154::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71340::DEBUG::2014-09-25 17:53:29,154::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71340::DEBUG::2014-09-25 17:53:29,154::task::993::Storage.TaskManager.Task::(_decref) Task=`34560840-bbbe-4789-9d7c-b4e7d8b16b31`::ref 0 aborting False >Thread-71340::DEBUG::2014-09-25 17:53:29,156::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:29,173::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:29,174::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71341::DEBUG::2014-09-25 17:53:29,176::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:29,932::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:29,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133626 s, 76.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:30,681::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:30,682::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:30,682::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:30,682::__init__::232::IOProcess::(_processLogs) (109760) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:30,683::__init__::232::IOProcess::(_processLogs) (109760) Queuing response >Thread-79::DEBUG::2014-09-25 17:53:30,684::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:53:30,693::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000424878 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:30,694::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:30,694::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:30,694::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:30,695::__init__::232::IOProcess::(_processLogs) (109761) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:30,695::__init__::232::IOProcess::(_processLogs) (109761) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:30,697::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:30,697::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:30,698::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:30,698::__init__::232::IOProcess::(_processLogs) (109762) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:30,698::__init__::232::IOProcess::(_processLogs) (109762) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:30,699::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:30,699::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:30,699::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:30,699::__init__::232::IOProcess::(_processLogs) (109763) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:30,700::__init__::232::IOProcess::(_processLogs) (109763) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:30,700::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:30,701::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:30,702::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:30,702::__init__::232::IOProcess::(_processLogs) (109764) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:30,702::__init__::232::IOProcess::(_processLogs) (109764) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:31,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:31,997::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0139614 s, 73.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:32,158::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71342::DEBUG::2014-09-25 17:53:32,159::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:53:32,159::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71342::DEBUG::2014-09-25 17:53:32,162::task::595::Storage.TaskManager.Task::(_updateState) Task=`c0d2c08f-f8d1-4085-8ebd-206a1ccf93fc`::moving from state init -> state preparing >Thread-71342::INFO::2014-09-25 17:53:32,162::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71342::INFO::2014-09-25 17:53:32,162::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71342::DEBUG::2014-09-25 17:53:32,163::task::1191::Storage.TaskManager.Task::(prepare) Task=`c0d2c08f-f8d1-4085-8ebd-206a1ccf93fc`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71342::DEBUG::2014-09-25 17:53:32,163::task::595::Storage.TaskManager.Task::(_updateState) Task=`c0d2c08f-f8d1-4085-8ebd-206a1ccf93fc`::moving from state preparing -> state finished >Thread-71342::DEBUG::2014-09-25 17:53:32,163::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71342::DEBUG::2014-09-25 17:53:32,163::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71342::DEBUG::2014-09-25 17:53:32,164::task::993::Storage.TaskManager.Task::(_decref) Task=`c0d2c08f-f8d1-4085-8ebd-206a1ccf93fc`::ref 0 aborting False >Thread-71342::DEBUG::2014-09-25 17:53:32,164::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71342::DEBUG::2014-09-25 17:53:32,164::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:32,175::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:32,176::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71343::DEBUG::2014-09-25 17:53:32,176::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71343::DEBUG::2014-09-25 17:53:32,178::task::595::Storage.TaskManager.Task::(_updateState) Task=`73c45099-46c5-48e2-a22f-a7cc8d1ab249`::moving from state init -> state preparing >Thread-71343::INFO::2014-09-25 17:53:32,178::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71343::DEBUG::2014-09-25 17:53:32,179::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`764b1f33-1c39-425f-a1c3-76151064f044`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71343::DEBUG::2014-09-25 17:53:32,179::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71343::DEBUG::2014-09-25 17:53:32,179::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71343::DEBUG::2014-09-25 17:53:32,179::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`764b1f33-1c39-425f-a1c3-76151064f044`::Granted request >Thread-71343::DEBUG::2014-09-25 17:53:32,180::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`73c45099-46c5-48e2-a22f-a7cc8d1ab249`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71343::DEBUG::2014-09-25 17:53:32,180::task::993::Storage.TaskManager.Task::(_decref) Task=`73c45099-46c5-48e2-a22f-a7cc8d1ab249`::ref 1 aborting False >Thread-71343::INFO::2014-09-25 17:53:32,185::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71343::DEBUG::2014-09-25 17:53:32,185::task::1191::Storage.TaskManager.Task::(prepare) Task=`73c45099-46c5-48e2-a22f-a7cc8d1ab249`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71343::DEBUG::2014-09-25 17:53:32,185::task::595::Storage.TaskManager.Task::(_updateState) Task=`73c45099-46c5-48e2-a22f-a7cc8d1ab249`::moving from state preparing -> state finished >Thread-71343::DEBUG::2014-09-25 17:53:32,185::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71343::DEBUG::2014-09-25 17:53:32,186::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71343::DEBUG::2014-09-25 17:53:32,186::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71343::DEBUG::2014-09-25 17:53:32,186::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71343::DEBUG::2014-09-25 17:53:32,187::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71343::DEBUG::2014-09-25 17:53:32,187::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71343::DEBUG::2014-09-25 17:53:32,187::task::993::Storage.TaskManager.Task::(_decref) Task=`73c45099-46c5-48e2-a22f-a7cc8d1ab249`::ref 0 aborting False >Thread-71343::DEBUG::2014-09-25 17:53:32,188::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71343::DEBUG::2014-09-25 17:53:32,188::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:32,224::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:32,225::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71344::DEBUG::2014-09-25 17:53:32,227::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:34,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:34,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135601 s, 75.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:35,019::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:35,019::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:35,019::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:35,019::__init__::232::IOProcess::(_processLogs) (109765) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:35,020::__init__::232::IOProcess::(_processLogs) (109765) Queuing response >Thread-81::DEBUG::2014-09-25 17:53:35,021::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:53:35,030::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00052342 s, 709 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:35,031::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:35,032::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:35,032::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:35,032::__init__::232::IOProcess::(_processLogs) (109766) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:35,034::__init__::232::IOProcess::(_processLogs) (109766) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:35,240::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:35,241::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71345::DEBUG::2014-09-25 17:53:35,243::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:36,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:36,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136087 s, 75.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:37,127::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:37,128::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:37,128::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:37,128::__init__::232::IOProcess::(_processLogs) (109767) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:37,129::__init__::232::IOProcess::(_processLogs) (109767) Queuing response >Thread-80::DEBUG::2014-09-25 17:53:37,130::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:53:37,139::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000482349 s, 784 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:37,140::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:37,140::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:37,141::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:37,141::__init__::232::IOProcess::(_processLogs) (109768) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:37,141::__init__::232::IOProcess::(_processLogs) (109768) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:38,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:38,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133254 s, 76.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:38,257::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:38,258::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71346::DEBUG::2014-09-25 17:53:38,260::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:40,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:40,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129212 s, 79.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:40,703::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:40,703::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:40,703::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:40,703::__init__::232::IOProcess::(_processLogs) (109769) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:40,704::__init__::232::IOProcess::(_processLogs) (109769) Queuing response >Thread-79::DEBUG::2014-09-25 17:53:40,705::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:53:40,714::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000445778 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:40,715::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:40,715::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:40,716::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:40,716::__init__::232::IOProcess::(_processLogs) (109770) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:40,716::__init__::232::IOProcess::(_processLogs) (109770) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:40,718::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:40,718::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:40,719::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:40,719::__init__::232::IOProcess::(_processLogs) (109771) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:40,719::__init__::232::IOProcess::(_processLogs) (109771) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:40,720::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:40,720::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:40,720::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:40,721::__init__::232::IOProcess::(_processLogs) (109772) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:40,721::__init__::232::IOProcess::(_processLogs) (109772) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:40,722::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:40,722::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:40,723::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:40,724::__init__::232::IOProcess::(_processLogs) (109773) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:40,724::__init__::232::IOProcess::(_processLogs) (109773) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:41,272::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:41,273::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71347::DEBUG::2014-09-25 17:53:41,275::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:42,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:42,210::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:42,213::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71348::DEBUG::2014-09-25 17:53:42,213::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71348::DEBUG::2014-09-25 17:53:42,215::task::595::Storage.TaskManager.Task::(_updateState) Task=`f9d5f684-c463-492f-a403-3c324586bf6f`::moving from state init -> state preparing >Thread-71348::INFO::2014-09-25 17:53:42,215::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71348::INFO::2014-09-25 17:53:42,216::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71348::DEBUG::2014-09-25 17:53:42,216::task::1191::Storage.TaskManager.Task::(prepare) Task=`f9d5f684-c463-492f-a403-3c324586bf6f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71348::DEBUG::2014-09-25 17:53:42,216::task::595::Storage.TaskManager.Task::(_updateState) Task=`f9d5f684-c463-492f-a403-3c324586bf6f`::moving from state preparing -> state finished >Thread-71348::DEBUG::2014-09-25 17:53:42,216::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71348::DEBUG::2014-09-25 17:53:42,217::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71348::DEBUG::2014-09-25 17:53:42,217::task::993::Storage.TaskManager.Task::(_decref) Task=`f9d5f684-c463-492f-a403-3c324586bf6f`::ref 0 aborting False >Thread-71348::DEBUG::2014-09-25 17:53:42,217::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71348::DEBUG::2014-09-25 17:53:42,218::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:42,228::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71349::DEBUG::2014-09-25 17:53:42,229::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:53:42,229::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71349::DEBUG::2014-09-25 17:53:42,231::task::595::Storage.TaskManager.Task::(_updateState) Task=`28d98b59-1755-4cba-8495-15d214a6b162`::moving from state init -> state preparing >Thread-71349::INFO::2014-09-25 17:53:42,231::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71349::DEBUG::2014-09-25 17:53:42,232::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ae4d1edd-50df-4a3c-9e05-516dd0e6d826`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71349::DEBUG::2014-09-25 17:53:42,232::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71349::DEBUG::2014-09-25 17:53:42,233::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71349::DEBUG::2014-09-25 17:53:42,233::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ae4d1edd-50df-4a3c-9e05-516dd0e6d826`::Granted request >Thread-71349::DEBUG::2014-09-25 17:53:42,233::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`28d98b59-1755-4cba-8495-15d214a6b162`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71349::DEBUG::2014-09-25 17:53:42,234::task::993::Storage.TaskManager.Task::(_decref) Task=`28d98b59-1755-4cba-8495-15d214a6b162`::ref 1 aborting False >Thread-71349::INFO::2014-09-25 17:53:42,238::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71349::DEBUG::2014-09-25 17:53:42,238::task::1191::Storage.TaskManager.Task::(prepare) Task=`28d98b59-1755-4cba-8495-15d214a6b162`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71349::DEBUG::2014-09-25 17:53:42,239::task::595::Storage.TaskManager.Task::(_updateState) Task=`28d98b59-1755-4cba-8495-15d214a6b162`::moving from state preparing -> state finished >Thread-71349::DEBUG::2014-09-25 17:53:42,239::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71349::DEBUG::2014-09-25 17:53:42,239::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71349::DEBUG::2014-09-25 17:53:42,239::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71349::DEBUG::2014-09-25 17:53:42,240::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71349::DEBUG::2014-09-25 17:53:42,240::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71349::DEBUG::2014-09-25 17:53:42,240::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71349::DEBUG::2014-09-25 17:53:42,241::task::993::Storage.TaskManager.Task::(_decref) Task=`28d98b59-1755-4cba-8495-15d214a6b162`::ref 0 aborting False >Thread-71349::DEBUG::2014-09-25 17:53:42,241::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71349::DEBUG::2014-09-25 17:53:42,241::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:42,464::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.265379 s, 3.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:44,285::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:44,286::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71350::DEBUG::2014-09-25 17:53:44,289::task::595::Storage.TaskManager.Task::(_updateState) Task=`51e32c63-f892-407b-83cf-d3e6f90fe2b4`::moving from state init -> state preparing >Thread-71350::INFO::2014-09-25 17:53:44,289::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71350::INFO::2014-09-25 17:53:44,289::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000445778', 'lastCheck': '3.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000482349', 'lastCheck': '7.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00052342', 'lastCheck': '9.3', 'valid': True}} >Thread-71350::DEBUG::2014-09-25 17:53:44,290::task::1191::Storage.TaskManager.Task::(prepare) Task=`51e32c63-f892-407b-83cf-d3e6f90fe2b4`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000445778', 'lastCheck': '3.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000482349', 'lastCheck': '7.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00052342', 'lastCheck': '9.3', 'valid': True}} >Thread-71350::DEBUG::2014-09-25 17:53:44,290::task::595::Storage.TaskManager.Task::(_updateState) Task=`51e32c63-f892-407b-83cf-d3e6f90fe2b4`::moving from state preparing -> state finished >Thread-71350::DEBUG::2014-09-25 17:53:44,290::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71350::DEBUG::2014-09-25 17:53:44,290::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71350::DEBUG::2014-09-25 17:53:44,291::task::993::Storage.TaskManager.Task::(_decref) Task=`51e32c63-f892-407b-83cf-d3e6f90fe2b4`::ref 0 aborting False >Thread-71350::DEBUG::2014-09-25 17:53:44,293::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:44,309::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:44,310::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71351::DEBUG::2014-09-25 17:53:44,312::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:44,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:44,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179831 s, 56.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:45,035::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:45,035::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:45,035::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:45,036::__init__::232::IOProcess::(_processLogs) (109774) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:45,036::__init__::232::IOProcess::(_processLogs) (109774) Queuing response >Thread-81::DEBUG::2014-09-25 17:53:45,037::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:53:45,046::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000440567 s, 842 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:45,047::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:45,048::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:45,048::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:45,050::__init__::232::IOProcess::(_processLogs) (109775) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:45,050::__init__::232::IOProcess::(_processLogs) (109775) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:46,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:46,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0274985 s, 37.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:47,144::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:47,145::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:47,145::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:47,145::__init__::232::IOProcess::(_processLogs) (109776) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:47,146::__init__::232::IOProcess::(_processLogs) (109776) Queuing response >Thread-80::DEBUG::2014-09-25 17:53:47,147::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:53:47,156::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000614639 s, 615 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:47,157::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:47,157::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:47,157::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:47,157::__init__::232::IOProcess::(_processLogs) (109777) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:47,158::__init__::232::IOProcess::(_processLogs) (109777) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:47,352::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:47,354::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71352::DEBUG::2014-09-25 17:53:47,355::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:48,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:48,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138379 s, 74.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:50,370::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:50,372::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71353::DEBUG::2014-09-25 17:53:50,374::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-79::DEBUG::2014-09-25 17:53:50,723::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:53:50,727::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,728::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,728::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,728::__init__::232::IOProcess::(_processLogs) (109778) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:53:50,729::__init__::232::IOProcess::(_processLogs) (109778) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:50,730::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,730::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,730::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,731::__init__::232::IOProcess::(_processLogs) (109779) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:53:50,731::__init__::232::IOProcess::(_processLogs) (109779) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:50,732::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,732::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,733::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,733::__init__::232::IOProcess::(_processLogs) (109780) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:53:50,733::__init__::232::IOProcess::(_processLogs) (109780) Queuing response >Thread-79::DEBUG::2014-09-25 17:53:50,738::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:53:50,738::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,739::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,739::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,739::__init__::232::IOProcess::(_processLogs) (109781) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:53:50,761::__init__::232::IOProcess::(_processLogs) (109781) Queuing response >Thread-79::DEBUG::2014-09-25 17:53:50,762::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:53:50,763::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,763::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,763::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,764::__init__::232::IOProcess::(_processLogs) (109782) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:53:50,764::__init__::232::IOProcess::(_processLogs) (109782) Queuing response >Thread-79::DEBUG::2014-09-25 17:53:50,765::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:53:50,766::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,767::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,767::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,767::__init__::232::IOProcess::(_processLogs) (109783) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:50,768::__init__::232::IOProcess::(_processLogs) (109783) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:50,768::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,768::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,769::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,769::__init__::232::IOProcess::(_processLogs) (109784) Got request for method 'glob' >Thread-79::DEBUG::2014-09-25 17:53:50,769::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:53:50,769::__init__::232::IOProcess::(_processLogs) (109784) Queuing response >Thread-79::INFO::2014-09-25 17:53:50,770::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:53:50,771::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:53:50,772::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,772::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,772::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,772::__init__::232::IOProcess::(_processLogs) (109785) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:50,773::__init__::232::IOProcess::(_processLogs) (109785) Queuing response >Thread-79::DEBUG::2014-09-25 17:53:50,774::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:53:50,783::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000436209 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:50,784::__init__::232::IOProcess::(_processLogs) Receiving request... >Dummy-148::DEBUG::2014-09-25 17:53:50,785::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:53:50,786::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,795::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,795::__init__::232::IOProcess::(_processLogs) (109786) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:50,796::__init__::232::IOProcess::(_processLogs) (109786) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:50,797::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,797::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,798::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,798::__init__::232::IOProcess::(_processLogs) (109787) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:50,805::__init__::232::IOProcess::(_processLogs) (109787) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:50,809::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,811::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,812::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,813::__init__::232::IOProcess::(_processLogs) (109788) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:50,815::__init__::232::IOProcess::(_processLogs) (109788) Queuing response >Thread-29::DEBUG::2014-09-25 17:53:50,817::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:50,819::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:50,820::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:50,821::__init__::232::IOProcess::(_processLogs) (109789) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:53:50,822::__init__::232::IOProcess::(_processLogs) (109789) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:50,825::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0290101 s, 35.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:52,267::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:52,269::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71357::DEBUG::2014-09-25 17:53:52,269::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71357::DEBUG::2014-09-25 17:53:52,271::task::595::Storage.TaskManager.Task::(_updateState) Task=`deb1cdcf-e7e1-4450-9273-5c5c762049b0`::moving from state init -> state preparing >Thread-71357::INFO::2014-09-25 17:53:52,271::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71357::INFO::2014-09-25 17:53:52,272::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71357::DEBUG::2014-09-25 17:53:52,272::task::1191::Storage.TaskManager.Task::(prepare) Task=`deb1cdcf-e7e1-4450-9273-5c5c762049b0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71357::DEBUG::2014-09-25 17:53:52,272::task::595::Storage.TaskManager.Task::(_updateState) Task=`deb1cdcf-e7e1-4450-9273-5c5c762049b0`::moving from state preparing -> state finished >Thread-71357::DEBUG::2014-09-25 17:53:52,272::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71357::DEBUG::2014-09-25 17:53:52,273::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71357::DEBUG::2014-09-25 17:53:52,273::task::993::Storage.TaskManager.Task::(_decref) Task=`deb1cdcf-e7e1-4450-9273-5c5c762049b0`::ref 0 aborting False >Thread-71357::DEBUG::2014-09-25 17:53:52,273::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71357::DEBUG::2014-09-25 17:53:52,274::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:52,288::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:52,289::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71358::DEBUG::2014-09-25 17:53:52,289::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71358::DEBUG::2014-09-25 17:53:52,291::task::595::Storage.TaskManager.Task::(_updateState) Task=`e63e036c-5183-4527-96ce-2c474ba44af2`::moving from state init -> state preparing >Thread-71358::INFO::2014-09-25 17:53:52,291::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71358::DEBUG::2014-09-25 17:53:52,292::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cc151265-9412-42e2-a9a1-e2cd05a85af9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71358::DEBUG::2014-09-25 17:53:52,292::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71358::DEBUG::2014-09-25 17:53:52,293::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71358::DEBUG::2014-09-25 17:53:52,293::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cc151265-9412-42e2-a9a1-e2cd05a85af9`::Granted request >Thread-71358::DEBUG::2014-09-25 17:53:52,293::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e63e036c-5183-4527-96ce-2c474ba44af2`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71358::DEBUG::2014-09-25 17:53:52,294::task::993::Storage.TaskManager.Task::(_decref) Task=`e63e036c-5183-4527-96ce-2c474ba44af2`::ref 1 aborting False >Thread-71358::INFO::2014-09-25 17:53:52,298::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71358::DEBUG::2014-09-25 17:53:52,298::task::1191::Storage.TaskManager.Task::(prepare) Task=`e63e036c-5183-4527-96ce-2c474ba44af2`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71358::DEBUG::2014-09-25 17:53:52,299::task::595::Storage.TaskManager.Task::(_updateState) Task=`e63e036c-5183-4527-96ce-2c474ba44af2`::moving from state preparing -> state finished >Thread-71358::DEBUG::2014-09-25 17:53:52,299::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71358::DEBUG::2014-09-25 17:53:52,299::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71358::DEBUG::2014-09-25 17:53:52,299::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71358::DEBUG::2014-09-25 17:53:52,300::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71358::DEBUG::2014-09-25 17:53:52,300::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71358::DEBUG::2014-09-25 17:53:52,300::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71358::DEBUG::2014-09-25 17:53:52,300::task::993::Storage.TaskManager.Task::(_decref) Task=`e63e036c-5183-4527-96ce-2c474ba44af2`::ref 0 aborting False >Thread-71358::DEBUG::2014-09-25 17:53:52,301::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71358::DEBUG::2014-09-25 17:53:52,301::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:52,845::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:52,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0177679 s, 57.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:53,387::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:53,388::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71359::DEBUG::2014-09-25 17:53:53,390::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:54,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:54,925::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0237422 s, 43.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:55,053::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:55,055::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:55,057::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:55,059::__init__::232::IOProcess::(_processLogs) (109790) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:55,062::__init__::232::IOProcess::(_processLogs) (109790) Queuing response >Thread-81::DEBUG::2014-09-25 17:53:55,065::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:53:55,077::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000377965 s, 982 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:55,081::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:55,084::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:55,086::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:55,088::__init__::232::IOProcess::(_processLogs) (109791) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:55,091::__init__::232::IOProcess::(_processLogs) (109791) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:56,404::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:56,405::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71360::DEBUG::2014-09-25 17:53:56,407::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:53:57,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:57,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136045 s, 75.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:57,161::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:57,162::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:57,163::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:57,164::__init__::232::IOProcess::(_processLogs) (109792) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:57,166::__init__::232::IOProcess::(_processLogs) (109792) Queuing response >Thread-80::DEBUG::2014-09-25 17:53:57,168::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:53:57,178::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000403644 s, 936 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:53:57,180::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:53:57,181::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:53:57,183::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:53:57,184::__init__::232::IOProcess::(_processLogs) (109793) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:53:57,185::__init__::232::IOProcess::(_processLogs) (109793) Queuing response >Dummy-148::DEBUG::2014-09-25 17:53:59,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:53:59,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133926 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:59,423::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:59,424::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71361::DEBUG::2014-09-25 17:53:59,427::task::595::Storage.TaskManager.Task::(_updateState) Task=`5c6bca18-64ea-40e5-99ea-cb9583d05441`::moving from state init -> state preparing >Thread-71361::INFO::2014-09-25 17:53:59,427::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71361::INFO::2014-09-25 17:53:59,428::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000436209', 'lastCheck': '8.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000403644', 'lastCheck': '2.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000377965', 'lastCheck': '4.3', 'valid': True}} >Thread-71361::DEBUG::2014-09-25 17:53:59,428::task::1191::Storage.TaskManager.Task::(prepare) Task=`5c6bca18-64ea-40e5-99ea-cb9583d05441`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000436209', 'lastCheck': '8.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000403644', 'lastCheck': '2.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000377965', 'lastCheck': '4.3', 'valid': True}} >Thread-71361::DEBUG::2014-09-25 17:53:59,428::task::595::Storage.TaskManager.Task::(_updateState) Task=`5c6bca18-64ea-40e5-99ea-cb9583d05441`::moving from state preparing -> state finished >Thread-71361::DEBUG::2014-09-25 17:53:59,428::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71361::DEBUG::2014-09-25 17:53:59,429::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71361::DEBUG::2014-09-25 17:53:59,429::task::993::Storage.TaskManager.Task::(_decref) Task=`5c6bca18-64ea-40e5-99ea-cb9583d05441`::ref 0 aborting False >Thread-71361::DEBUG::2014-09-25 17:53:59,431::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:53:59,458::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:53:59,459::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71362::DEBUG::2014-09-25 17:53:59,461::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:54:00,825::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:00,826::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:00,826::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:00,826::__init__::232::IOProcess::(_processLogs) (109794) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:00,827::__init__::232::IOProcess::(_processLogs) (109794) Queuing response >Thread-79::DEBUG::2014-09-25 17:54:00,828::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:54:00,837::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000321333 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:00,838::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:00,839::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:00,839::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:00,839::__init__::232::IOProcess::(_processLogs) (109795) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:00,839::__init__::232::IOProcess::(_processLogs) (109795) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:00,840::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:00,841::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:00,842::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:00,842::__init__::232::IOProcess::(_processLogs) (109796) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:00,842::__init__::232::IOProcess::(_processLogs) (109796) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:00,843::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:00,843::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:00,844::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:00,844::__init__::232::IOProcess::(_processLogs) (109797) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:00,844::__init__::232::IOProcess::(_processLogs) (109797) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:00,845::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:00,845::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:00,846::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:00,846::__init__::232::IOProcess::(_processLogs) (109798) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:00,847::__init__::232::IOProcess::(_processLogs) (109798) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:01,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:01,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134859 s, 75.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:02,339::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:02,341::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71363::DEBUG::2014-09-25 17:54:02,341::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71363::DEBUG::2014-09-25 17:54:02,343::task::595::Storage.TaskManager.Task::(_updateState) Task=`b164217c-5ff5-414a-a76d-4630c21a8b0a`::moving from state init -> state preparing >Thread-71363::INFO::2014-09-25 17:54:02,343::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71363::INFO::2014-09-25 17:54:02,344::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71363::DEBUG::2014-09-25 17:54:02,344::task::1191::Storage.TaskManager.Task::(prepare) Task=`b164217c-5ff5-414a-a76d-4630c21a8b0a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71363::DEBUG::2014-09-25 17:54:02,344::task::595::Storage.TaskManager.Task::(_updateState) Task=`b164217c-5ff5-414a-a76d-4630c21a8b0a`::moving from state preparing -> state finished >Thread-71363::DEBUG::2014-09-25 17:54:02,344::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71363::DEBUG::2014-09-25 17:54:02,345::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71363::DEBUG::2014-09-25 17:54:02,345::task::993::Storage.TaskManager.Task::(_decref) Task=`b164217c-5ff5-414a-a76d-4630c21a8b0a`::ref 0 aborting False >Thread-71363::DEBUG::2014-09-25 17:54:02,345::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71363::DEBUG::2014-09-25 17:54:02,346::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:02,357::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:02,358::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71364::DEBUG::2014-09-25 17:54:02,358::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71364::DEBUG::2014-09-25 17:54:02,359::task::595::Storage.TaskManager.Task::(_updateState) Task=`1afa7ecb-4b38-4c9d-9893-f45f505722ff`::moving from state init -> state preparing >Thread-71364::INFO::2014-09-25 17:54:02,359::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71364::DEBUG::2014-09-25 17:54:02,360::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`248166af-924b-4d51-8732-bb2ca127f6f5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71364::DEBUG::2014-09-25 17:54:02,360::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71364::DEBUG::2014-09-25 17:54:02,361::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71364::DEBUG::2014-09-25 17:54:02,361::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`248166af-924b-4d51-8732-bb2ca127f6f5`::Granted request >Thread-71364::DEBUG::2014-09-25 17:54:02,362::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1afa7ecb-4b38-4c9d-9893-f45f505722ff`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71364::DEBUG::2014-09-25 17:54:02,362::task::993::Storage.TaskManager.Task::(_decref) Task=`1afa7ecb-4b38-4c9d-9893-f45f505722ff`::ref 1 aborting False >Thread-71364::INFO::2014-09-25 17:54:02,366::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71364::DEBUG::2014-09-25 17:54:02,366::task::1191::Storage.TaskManager.Task::(prepare) Task=`1afa7ecb-4b38-4c9d-9893-f45f505722ff`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71364::DEBUG::2014-09-25 17:54:02,367::task::595::Storage.TaskManager.Task::(_updateState) Task=`1afa7ecb-4b38-4c9d-9893-f45f505722ff`::moving from state preparing -> state finished >Thread-71364::DEBUG::2014-09-25 17:54:02,367::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71364::DEBUG::2014-09-25 17:54:02,367::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71364::DEBUG::2014-09-25 17:54:02,368::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71364::DEBUG::2014-09-25 17:54:02,368::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71364::DEBUG::2014-09-25 17:54:02,368::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71364::DEBUG::2014-09-25 17:54:02,368::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71364::DEBUG::2014-09-25 17:54:02,369::task::993::Storage.TaskManager.Task::(_decref) Task=`1afa7ecb-4b38-4c9d-9893-f45f505722ff`::ref 0 aborting False >Thread-71364::DEBUG::2014-09-25 17:54:02,369::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71364::DEBUG::2014-09-25 17:54:02,370::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:02,520::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:02,521::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71365::DEBUG::2014-09-25 17:54:02,522::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:03,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:03,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0296651 s, 34.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:05,094::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:05,094::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:05,095::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:05,095::__init__::232::IOProcess::(_processLogs) (109799) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:05,105::__init__::232::IOProcess::(_processLogs) (109799) Queuing response >Thread-81::DEBUG::2014-09-25 17:54:05,106::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:54:05,136::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00922548 s, 40.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:05,137::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:05,138::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:05,140::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:05,142::__init__::232::IOProcess::(_processLogs) (109800) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:05,144::__init__::232::IOProcess::(_processLogs) (109800) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:05,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:05,379::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181853 s, 56.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:05,536::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:05,538::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71366::DEBUG::2014-09-25 17:54:05,540::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:54:07,188::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:07,189::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:07,189::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:07,189::__init__::232::IOProcess::(_processLogs) (109801) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:07,189::__init__::232::IOProcess::(_processLogs) (109801) Queuing response >Thread-80::DEBUG::2014-09-25 17:54:07,190::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:54:07,199::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000393214 s, 961 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:07,201::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:07,201::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:07,202::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:07,202::__init__::232::IOProcess::(_processLogs) (109802) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:07,202::__init__::232::IOProcess::(_processLogs) (109802) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:07,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:07,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137193 s, 74.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:08,554::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:08,555::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71367::DEBUG::2014-09-25 17:54:08,557::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:09,441::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:09,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135155 s, 75.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:10,847::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:10,847::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:10,848::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:10,848::__init__::232::IOProcess::(_processLogs) (109803) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:10,848::__init__::232::IOProcess::(_processLogs) (109803) Queuing response >Thread-79::DEBUG::2014-09-25 17:54:10,850::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:54:10,859::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000357561 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:10,860::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:10,861::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:10,862::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:10,862::__init__::232::IOProcess::(_processLogs) (109804) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:10,862::__init__::232::IOProcess::(_processLogs) (109804) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:10,863::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:10,863::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:10,863::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:10,864::__init__::232::IOProcess::(_processLogs) (109805) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:10,864::__init__::232::IOProcess::(_processLogs) (109805) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:10,865::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:10,865::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:10,865::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:10,865::__init__::232::IOProcess::(_processLogs) (109806) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:10,866::__init__::232::IOProcess::(_processLogs) (109806) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:10,867::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:10,867::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:10,867::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:10,868::__init__::232::IOProcess::(_processLogs) (109807) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:10,868::__init__::232::IOProcess::(_processLogs) (109807) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:11,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:11,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013265 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:11,570::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:11,572::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71368::DEBUG::2014-09-25 17:54:11,574::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:12,398::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:12,400::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71369::DEBUG::2014-09-25 17:54:12,400::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71369::DEBUG::2014-09-25 17:54:12,402::task::595::Storage.TaskManager.Task::(_updateState) Task=`28963896-8bdf-43ef-a6bd-2a984d9a0916`::moving from state init -> state preparing >Thread-71369::INFO::2014-09-25 17:54:12,403::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71369::INFO::2014-09-25 17:54:12,403::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71369::DEBUG::2014-09-25 17:54:12,403::task::1191::Storage.TaskManager.Task::(prepare) Task=`28963896-8bdf-43ef-a6bd-2a984d9a0916`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71369::DEBUG::2014-09-25 17:54:12,404::task::595::Storage.TaskManager.Task::(_updateState) Task=`28963896-8bdf-43ef-a6bd-2a984d9a0916`::moving from state preparing -> state finished >Thread-71369::DEBUG::2014-09-25 17:54:12,404::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71369::DEBUG::2014-09-25 17:54:12,404::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71369::DEBUG::2014-09-25 17:54:12,404::task::993::Storage.TaskManager.Task::(_decref) Task=`28963896-8bdf-43ef-a6bd-2a984d9a0916`::ref 0 aborting False >Thread-71369::DEBUG::2014-09-25 17:54:12,405::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71369::DEBUG::2014-09-25 17:54:12,405::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:12,417::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:12,418::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71370::DEBUG::2014-09-25 17:54:12,418::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71370::DEBUG::2014-09-25 17:54:12,420::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5cfaf99-ff75-415a-9bab-66d5b6c1fa95`::moving from state init -> state preparing >Thread-71370::INFO::2014-09-25 17:54:12,420::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71370::DEBUG::2014-09-25 17:54:12,421::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f13c860c-206e-4498-9732-2d826517795e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71370::DEBUG::2014-09-25 17:54:12,421::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71370::DEBUG::2014-09-25 17:54:12,422::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71370::DEBUG::2014-09-25 17:54:12,422::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f13c860c-206e-4498-9732-2d826517795e`::Granted request >Thread-71370::DEBUG::2014-09-25 17:54:12,423::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a5cfaf99-ff75-415a-9bab-66d5b6c1fa95`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71370::DEBUG::2014-09-25 17:54:12,423::task::993::Storage.TaskManager.Task::(_decref) Task=`a5cfaf99-ff75-415a-9bab-66d5b6c1fa95`::ref 1 aborting False >Thread-71370::INFO::2014-09-25 17:54:12,427::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71370::DEBUG::2014-09-25 17:54:12,428::task::1191::Storage.TaskManager.Task::(prepare) Task=`a5cfaf99-ff75-415a-9bab-66d5b6c1fa95`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71370::DEBUG::2014-09-25 17:54:12,428::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5cfaf99-ff75-415a-9bab-66d5b6c1fa95`::moving from state preparing -> state finished >Thread-71370::DEBUG::2014-09-25 17:54:12,428::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71370::DEBUG::2014-09-25 17:54:12,428::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71370::DEBUG::2014-09-25 17:54:12,429::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71370::DEBUG::2014-09-25 17:54:12,429::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71370::DEBUG::2014-09-25 17:54:12,429::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71370::DEBUG::2014-09-25 17:54:12,429::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71370::DEBUG::2014-09-25 17:54:12,430::task::993::Storage.TaskManager.Task::(_decref) Task=`a5cfaf99-ff75-415a-9bab-66d5b6c1fa95`::ref 0 aborting False >Thread-71370::DEBUG::2014-09-25 17:54:12,430::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71370::DEBUG::2014-09-25 17:54:12,431::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:13,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:13,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133807 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:14,588::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:14,589::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71371::DEBUG::2014-09-25 17:54:14,592::task::595::Storage.TaskManager.Task::(_updateState) Task=`c73dd22b-e818-421b-8cc6-c6fdbc8c1016`::moving from state init -> state preparing >Thread-71371::INFO::2014-09-25 17:54:14,592::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71371::INFO::2014-09-25 17:54:14,593::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000357561', 'lastCheck': '3.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000393214', 'lastCheck': '7.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00922548', 'lastCheck': '9.4', 'valid': True}} >Thread-71371::DEBUG::2014-09-25 17:54:14,593::task::1191::Storage.TaskManager.Task::(prepare) Task=`c73dd22b-e818-421b-8cc6-c6fdbc8c1016`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000357561', 'lastCheck': '3.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000393214', 'lastCheck': '7.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00922548', 'lastCheck': '9.4', 'valid': True}} >Thread-71371::DEBUG::2014-09-25 17:54:14,593::task::595::Storage.TaskManager.Task::(_updateState) Task=`c73dd22b-e818-421b-8cc6-c6fdbc8c1016`::moving from state preparing -> state finished >Thread-71371::DEBUG::2014-09-25 17:54:14,593::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71371::DEBUG::2014-09-25 17:54:14,594::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71371::DEBUG::2014-09-25 17:54:14,594::task::993::Storage.TaskManager.Task::(_decref) Task=`c73dd22b-e818-421b-8cc6-c6fdbc8c1016`::ref 0 aborting False >Thread-71371::DEBUG::2014-09-25 17:54:14,596::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:14,619::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:14,620::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71372::DEBUG::2014-09-25 17:54:14,622::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:54:15,147::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:15,147::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:15,148::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:54:15,149::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:54:15,149::__init__::232::IOProcess::(_processLogs) (109808) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:15,158::__init__::232::IOProcess::(_processLogs) (109808) Queuing response >Thread-81::DEBUG::2014-09-25 17:54:15,159::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000456377 s, 813 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:15,160::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:15,160::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:15,160::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:15,161::__init__::232::IOProcess::(_processLogs) (109809) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:15,161::__init__::232::IOProcess::(_processLogs) (109809) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:15,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:15,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134528 s, 76.1 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 17:54:17,203::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:54:17,206::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:17,207::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:17,207::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:17,207::__init__::232::IOProcess::(_processLogs) (109810) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:54:17,207::__init__::232::IOProcess::(_processLogs) (109810) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:17,208::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:17,208::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:17,208::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:17,208::__init__::232::IOProcess::(_processLogs) (109811) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:54:17,209::__init__::232::IOProcess::(_processLogs) (109811) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:17,211::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:17,212::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:17,213::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:17,214::__init__::232::IOProcess::(_processLogs) (109812) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:54:17,215::__init__::232::IOProcess::(_processLogs) (109812) Queuing response >Thread-80::DEBUG::2014-09-25 17:54:17,216::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:54:17,217::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:17,217::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:17,217::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:17,217::__init__::232::IOProcess::(_processLogs) (109813) Got request for method 'touch' >Thread-80::DEBUG::2014-09-25 17:54:17,233::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:54:17,233::__init__::232::IOProcess::(_processLogs) (109813) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:17,234::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:17,235::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:17,235::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:17,235::__init__::232::IOProcess::(_processLogs) (109814) Got request for method 'readfile' >Thread-80::DEBUG::2014-09-25 17:54:17,236::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:54:17,236::__init__::232::IOProcess::(_processLogs) (109814) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:17,237::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:17,238::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:17,238::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:17,238::__init__::232::IOProcess::(_processLogs) (109815) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:17,238::__init__::232::IOProcess::(_processLogs) (109815) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:17,238::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:17,239::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:17,239::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:17,239::__init__::232::IOProcess::(_processLogs) (109816) Got request for method 'glob' >Thread-80::DEBUG::2014-09-25 17:54:17,239::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:54:17,239::__init__::232::IOProcess::(_processLogs) (109816) Queuing response >Thread-80::INFO::2014-09-25 17:54:17,240::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:54:17,240::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:54:17,241::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:17,241::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:17,241::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:17,241::__init__::232::IOProcess::(_processLogs) (109817) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:17,242::__init__::232::IOProcess::(_processLogs) (109817) Queuing response >Thread-80::DEBUG::2014-09-25 17:54:17,242::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:54:17,250::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000409912 s, 922 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:17,250::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:17,251::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:17,251::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:17,251::__init__::232::IOProcess::(_processLogs) (109818) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:17,251::__init__::232::IOProcess::(_processLogs) (109818) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:17,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:17,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136633 s, 74.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:17,669::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:17,671::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71376::DEBUG::2014-09-25 17:54:17,673::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:19,652::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:19,675::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135492 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:20,685::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:20,686::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71377::DEBUG::2014-09-25 17:54:20,688::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:54:20,870::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:20,870::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:20,870::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:20,871::__init__::232::IOProcess::(_processLogs) (109819) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:20,871::__init__::232::IOProcess::(_processLogs) (109819) Queuing response >Thread-79::DEBUG::2014-09-25 17:54:20,872::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:54:20,881::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000321897 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:20,882::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:20,883::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:20,883::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:20,883::__init__::232::IOProcess::(_processLogs) (109820) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:20,884::__init__::232::IOProcess::(_processLogs) (109820) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:20,885::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:20,886::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:20,887::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:20,887::__init__::232::IOProcess::(_processLogs) (109821) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:20,887::__init__::232::IOProcess::(_processLogs) (109821) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:20,888::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:20,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:20,888::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:20,889::__init__::232::IOProcess::(_processLogs) (109822) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:20,889::__init__::232::IOProcess::(_processLogs) (109822) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:20,890::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:20,890::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:20,891::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:20,892::__init__::232::IOProcess::(_processLogs) (109823) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:20,892::__init__::232::IOProcess::(_processLogs) (109823) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:21,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:21,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134768 s, 76.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:22,458::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71378::DEBUG::2014-09-25 17:54:22,460::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:54:22,460::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71378::DEBUG::2014-09-25 17:54:22,463::task::595::Storage.TaskManager.Task::(_updateState) Task=`b1e85a4c-2d62-4bb1-b05e-aa3924e14a22`::moving from state init -> state preparing >Thread-71378::INFO::2014-09-25 17:54:22,463::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71378::INFO::2014-09-25 17:54:22,464::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71378::DEBUG::2014-09-25 17:54:22,464::task::1191::Storage.TaskManager.Task::(prepare) Task=`b1e85a4c-2d62-4bb1-b05e-aa3924e14a22`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71378::DEBUG::2014-09-25 17:54:22,464::task::595::Storage.TaskManager.Task::(_updateState) Task=`b1e85a4c-2d62-4bb1-b05e-aa3924e14a22`::moving from state preparing -> state finished >Thread-71378::DEBUG::2014-09-25 17:54:22,464::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71378::DEBUG::2014-09-25 17:54:22,465::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71378::DEBUG::2014-09-25 17:54:22,465::task::993::Storage.TaskManager.Task::(_decref) Task=`b1e85a4c-2d62-4bb1-b05e-aa3924e14a22`::ref 0 aborting False >Thread-71378::DEBUG::2014-09-25 17:54:22,465::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71378::DEBUG::2014-09-25 17:54:22,465::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:22,480::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:22,481::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71379::DEBUG::2014-09-25 17:54:22,481::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71379::DEBUG::2014-09-25 17:54:22,483::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2b940e7-8b6d-4293-a80d-59b914cbe1a4`::moving from state init -> state preparing >Thread-71379::INFO::2014-09-25 17:54:22,483::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71379::DEBUG::2014-09-25 17:54:22,484::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7fbb1bab-3c7b-45d0-ad17-985315fadfee`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71379::DEBUG::2014-09-25 17:54:22,484::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71379::DEBUG::2014-09-25 17:54:22,484::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71379::DEBUG::2014-09-25 17:54:22,484::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7fbb1bab-3c7b-45d0-ad17-985315fadfee`::Granted request >Thread-71379::DEBUG::2014-09-25 17:54:22,485::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`d2b940e7-8b6d-4293-a80d-59b914cbe1a4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71379::DEBUG::2014-09-25 17:54:22,485::task::993::Storage.TaskManager.Task::(_decref) Task=`d2b940e7-8b6d-4293-a80d-59b914cbe1a4`::ref 1 aborting False >Thread-71379::INFO::2014-09-25 17:54:22,490::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71379::DEBUG::2014-09-25 17:54:22,490::task::1191::Storage.TaskManager.Task::(prepare) Task=`d2b940e7-8b6d-4293-a80d-59b914cbe1a4`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71379::DEBUG::2014-09-25 17:54:22,490::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2b940e7-8b6d-4293-a80d-59b914cbe1a4`::moving from state preparing -> state finished >Thread-71379::DEBUG::2014-09-25 17:54:22,490::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71379::DEBUG::2014-09-25 17:54:22,491::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71379::DEBUG::2014-09-25 17:54:22,491::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71379::DEBUG::2014-09-25 17:54:22,491::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71379::DEBUG::2014-09-25 17:54:22,492::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71379::DEBUG::2014-09-25 17:54:22,492::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71379::DEBUG::2014-09-25 17:54:22,492::task::993::Storage.TaskManager.Task::(_decref) Task=`d2b940e7-8b6d-4293-a80d-59b914cbe1a4`::ref 0 aborting False >Thread-71379::DEBUG::2014-09-25 17:54:22,493::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71379::DEBUG::2014-09-25 17:54:22,493::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:23,700::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:23,702::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71380::DEBUG::2014-09-25 17:54:23,704::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:23,737::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:23,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148165 s, 69.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:25,163::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:25,164::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:25,164::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:25,164::__init__::232::IOProcess::(_processLogs) (109824) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:25,165::__init__::232::IOProcess::(_processLogs) (109824) Queuing response >Thread-81::DEBUG::2014-09-25 17:54:25,165::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:54:25,174::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000361988 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:25,175::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:25,176::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:25,176::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:25,176::__init__::232::IOProcess::(_processLogs) (109825) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:25,177::__init__::232::IOProcess::(_processLogs) (109825) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:25,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:25,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131891 s, 77.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:26,714::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:26,716::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71381::DEBUG::2014-09-25 17:54:26,718::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:54:27,253::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:27,254::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:27,254::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:27,254::__init__::232::IOProcess::(_processLogs) (109826) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:27,254::__init__::232::IOProcess::(_processLogs) (109826) Queuing response >Thread-80::DEBUG::2014-09-25 17:54:27,255::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:54:27,264::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00044579 s, 848 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:27,265::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:27,266::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:27,267::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:27,267::__init__::232::IOProcess::(_processLogs) (109827) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:27,268::__init__::232::IOProcess::(_processLogs) (109827) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:27,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:27,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0490711 s, 20.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:29,729::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:29,731::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71382::DEBUG::2014-09-25 17:54:29,733::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb0bfcf2-1802-4950-9926-5f2998de75f0`::moving from state init -> state preparing >Thread-71382::INFO::2014-09-25 17:54:29,733::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71382::INFO::2014-09-25 17:54:29,734::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000321897', 'lastCheck': '8.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00044579', 'lastCheck': '2.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000361988', 'lastCheck': '4.6', 'valid': True}} >Thread-71382::DEBUG::2014-09-25 17:54:29,734::task::1191::Storage.TaskManager.Task::(prepare) Task=`fb0bfcf2-1802-4950-9926-5f2998de75f0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000321897', 'lastCheck': '8.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00044579', 'lastCheck': '2.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000361988', 'lastCheck': '4.6', 'valid': True}} >Thread-71382::DEBUG::2014-09-25 17:54:29,734::task::595::Storage.TaskManager.Task::(_updateState) Task=`fb0bfcf2-1802-4950-9926-5f2998de75f0`::moving from state preparing -> state finished >Thread-71382::DEBUG::2014-09-25 17:54:29,735::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71382::DEBUG::2014-09-25 17:54:29,735::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71382::DEBUG::2014-09-25 17:54:29,735::task::993::Storage.TaskManager.Task::(_decref) Task=`fb0bfcf2-1802-4950-9926-5f2998de75f0`::ref 0 aborting False >Thread-71382::DEBUG::2014-09-25 17:54:29,737::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:29,753::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:29,754::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71383::DEBUG::2014-09-25 17:54:29,756::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:29,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:29,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133168 s, 76.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:30,892::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:30,893::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:30,893::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:30,893::__init__::232::IOProcess::(_processLogs) (109828) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:30,893::__init__::232::IOProcess::(_processLogs) (109828) Queuing response >Thread-79::DEBUG::2014-09-25 17:54:30,894::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:54:30,903::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000372026 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:30,904::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:30,904::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:30,904::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:30,905::__init__::232::IOProcess::(_processLogs) (109829) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:30,905::__init__::232::IOProcess::(_processLogs) (109829) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:30,907::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:30,907::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:30,908::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:30,908::__init__::232::IOProcess::(_processLogs) (109830) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:30,908::__init__::232::IOProcess::(_processLogs) (109830) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:30,909::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:30,909::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:30,910::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:30,910::__init__::232::IOProcess::(_processLogs) (109831) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:30,910::__init__::232::IOProcess::(_processLogs) (109831) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:30,911::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:30,911::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:30,912::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:30,912::__init__::232::IOProcess::(_processLogs) (109832) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:30,913::__init__::232::IOProcess::(_processLogs) (109832) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:31,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:31,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133092 s, 76.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:32,527::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71384::DEBUG::2014-09-25 17:54:32,528::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:54:32,528::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71384::DEBUG::2014-09-25 17:54:32,531::task::595::Storage.TaskManager.Task::(_updateState) Task=`c508920f-ccd0-4c27-aadd-658befc6eaca`::moving from state init -> state preparing >Thread-71384::INFO::2014-09-25 17:54:32,531::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71384::INFO::2014-09-25 17:54:32,532::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71384::DEBUG::2014-09-25 17:54:32,532::task::1191::Storage.TaskManager.Task::(prepare) Task=`c508920f-ccd0-4c27-aadd-658befc6eaca`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71384::DEBUG::2014-09-25 17:54:32,532::task::595::Storage.TaskManager.Task::(_updateState) Task=`c508920f-ccd0-4c27-aadd-658befc6eaca`::moving from state preparing -> state finished >Thread-71384::DEBUG::2014-09-25 17:54:32,532::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71384::DEBUG::2014-09-25 17:54:32,532::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71384::DEBUG::2014-09-25 17:54:32,533::task::993::Storage.TaskManager.Task::(_decref) Task=`c508920f-ccd0-4c27-aadd-658befc6eaca`::ref 0 aborting False >Thread-71384::DEBUG::2014-09-25 17:54:32,533::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71384::DEBUG::2014-09-25 17:54:32,533::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:32,544::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71385::DEBUG::2014-09-25 17:54:32,545::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:54:32,545::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71385::DEBUG::2014-09-25 17:54:32,548::task::595::Storage.TaskManager.Task::(_updateState) Task=`b87cc6a0-fc8a-426b-805f-7f13e7a5c92f`::moving from state init -> state preparing >Thread-71385::INFO::2014-09-25 17:54:32,549::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71385::DEBUG::2014-09-25 17:54:32,549::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a58f5894-2e1d-4e4f-acd1-4f6305fbd8fc`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71385::DEBUG::2014-09-25 17:54:32,550::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71385::DEBUG::2014-09-25 17:54:32,550::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71385::DEBUG::2014-09-25 17:54:32,550::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a58f5894-2e1d-4e4f-acd1-4f6305fbd8fc`::Granted request >Thread-71385::DEBUG::2014-09-25 17:54:32,551::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b87cc6a0-fc8a-426b-805f-7f13e7a5c92f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71385::DEBUG::2014-09-25 17:54:32,551::task::993::Storage.TaskManager.Task::(_decref) Task=`b87cc6a0-fc8a-426b-805f-7f13e7a5c92f`::ref 1 aborting False >Thread-71385::INFO::2014-09-25 17:54:32,555::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71385::DEBUG::2014-09-25 17:54:32,556::task::1191::Storage.TaskManager.Task::(prepare) Task=`b87cc6a0-fc8a-426b-805f-7f13e7a5c92f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71385::DEBUG::2014-09-25 17:54:32,556::task::595::Storage.TaskManager.Task::(_updateState) Task=`b87cc6a0-fc8a-426b-805f-7f13e7a5c92f`::moving from state preparing -> state finished >Thread-71385::DEBUG::2014-09-25 17:54:32,556::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71385::DEBUG::2014-09-25 17:54:32,556::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71385::DEBUG::2014-09-25 17:54:32,557::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71385::DEBUG::2014-09-25 17:54:32,557::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71385::DEBUG::2014-09-25 17:54:32,557::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71385::DEBUG::2014-09-25 17:54:32,558::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71385::DEBUG::2014-09-25 17:54:32,558::task::993::Storage.TaskManager.Task::(_decref) Task=`b87cc6a0-fc8a-426b-805f-7f13e7a5c92f`::ref 0 aborting False >Thread-71385::DEBUG::2014-09-25 17:54:32,558::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71385::DEBUG::2014-09-25 17:54:32,559::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:32,799::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:32,800::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71386::DEBUG::2014-09-25 17:54:32,802::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:33,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:34,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129342 s, 79.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:35,179::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:35,180::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:35,180::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:35,180::__init__::232::IOProcess::(_processLogs) (109833) Got request for method 'statvfs' >Thread-81::DEBUG::2014-09-25 17:54:35,182::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:54:35,182::__init__::232::IOProcess::(_processLogs) (109833) Queuing response >Thread-81::DEBUG::2014-09-25 17:54:35,191::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000390926 s, 949 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:35,192::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:35,193::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:35,193::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:35,193::__init__::232::IOProcess::(_processLogs) (109834) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:35,194::__init__::232::IOProcess::(_processLogs) (109834) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:35,812::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:35,813::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71387::DEBUG::2014-09-25 17:54:35,815::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:36,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:36,048::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134475 s, 76.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:37,268::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:37,269::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:37,269::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:37,269::__init__::232::IOProcess::(_processLogs) (109835) Got request for method 'statvfs' >Thread-80::DEBUG::2014-09-25 17:54:37,270::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:54:37,271::__init__::232::IOProcess::(_processLogs) (109835) Queuing response >Thread-80::DEBUG::2014-09-25 17:54:37,280::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000422796 s, 894 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:37,281::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:37,282::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:37,283::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:37,284::__init__::232::IOProcess::(_processLogs) (109836) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:37,284::__init__::232::IOProcess::(_processLogs) (109836) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:38,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:38,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180943 s, 56.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:38,829::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:38,830::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71388::DEBUG::2014-09-25 17:54:38,833::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:40,129::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:40,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131911 s, 77.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:40,913::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:40,914::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:40,914::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:54:40,915::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:54:40,915::__init__::232::IOProcess::(_processLogs) (109837) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:40,924::__init__::232::IOProcess::(_processLogs) (109837) Queuing response >Thread-79::DEBUG::2014-09-25 17:54:40,925::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000425637 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:40,926::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:40,927::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:40,927::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:40,927::__init__::232::IOProcess::(_processLogs) (109838) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:40,928::__init__::232::IOProcess::(_processLogs) (109838) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:40,929::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:40,930::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:40,930::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:40,930::__init__::232::IOProcess::(_processLogs) (109839) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:40,931::__init__::232::IOProcess::(_processLogs) (109839) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:40,931::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:40,932::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:40,932::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:40,932::__init__::232::IOProcess::(_processLogs) (109840) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:40,933::__init__::232::IOProcess::(_processLogs) (109840) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:40,933::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:40,934::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:40,935::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:40,935::__init__::232::IOProcess::(_processLogs) (109841) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:40,935::__init__::232::IOProcess::(_processLogs) (109841) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:41,847::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:41,848::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71389::DEBUG::2014-09-25 17:54:41,850::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:42,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:42,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192041 s, 53.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:42,584::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71390::DEBUG::2014-09-25 17:54:42,585::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:54:42,586::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71390::DEBUG::2014-09-25 17:54:42,588::task::595::Storage.TaskManager.Task::(_updateState) Task=`20ca3282-c8f0-4548-bd8f-77596aa99a4c`::moving from state init -> state preparing >Thread-71390::INFO::2014-09-25 17:54:42,588::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71390::INFO::2014-09-25 17:54:42,589::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71390::DEBUG::2014-09-25 17:54:42,589::task::1191::Storage.TaskManager.Task::(prepare) Task=`20ca3282-c8f0-4548-bd8f-77596aa99a4c`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71390::DEBUG::2014-09-25 17:54:42,589::task::595::Storage.TaskManager.Task::(_updateState) Task=`20ca3282-c8f0-4548-bd8f-77596aa99a4c`::moving from state preparing -> state finished >Thread-71390::DEBUG::2014-09-25 17:54:42,590::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71390::DEBUG::2014-09-25 17:54:42,590::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71390::DEBUG::2014-09-25 17:54:42,590::task::993::Storage.TaskManager.Task::(_decref) Task=`20ca3282-c8f0-4548-bd8f-77596aa99a4c`::ref 0 aborting False >Thread-71390::DEBUG::2014-09-25 17:54:42,590::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71390::DEBUG::2014-09-25 17:54:42,591::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:42,606::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71391::DEBUG::2014-09-25 17:54:42,607::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:54:42,607::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71391::DEBUG::2014-09-25 17:54:42,610::task::595::Storage.TaskManager.Task::(_updateState) Task=`c6ef6716-c3e2-497f-9545-ccb924eb1429`::moving from state init -> state preparing >Thread-71391::INFO::2014-09-25 17:54:42,610::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71391::DEBUG::2014-09-25 17:54:42,611::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9baf3f06-2435-46ff-8e3f-2a0350f756a3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71391::DEBUG::2014-09-25 17:54:42,611::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71391::DEBUG::2014-09-25 17:54:42,611::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71391::DEBUG::2014-09-25 17:54:42,611::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9baf3f06-2435-46ff-8e3f-2a0350f756a3`::Granted request >Thread-71391::DEBUG::2014-09-25 17:54:42,612::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c6ef6716-c3e2-497f-9545-ccb924eb1429`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71391::DEBUG::2014-09-25 17:54:42,612::task::993::Storage.TaskManager.Task::(_decref) Task=`c6ef6716-c3e2-497f-9545-ccb924eb1429`::ref 1 aborting False >Thread-71391::INFO::2014-09-25 17:54:42,617::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71391::DEBUG::2014-09-25 17:54:42,617::task::1191::Storage.TaskManager.Task::(prepare) Task=`c6ef6716-c3e2-497f-9545-ccb924eb1429`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71391::DEBUG::2014-09-25 17:54:42,617::task::595::Storage.TaskManager.Task::(_updateState) Task=`c6ef6716-c3e2-497f-9545-ccb924eb1429`::moving from state preparing -> state finished >Thread-71391::DEBUG::2014-09-25 17:54:42,617::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71391::DEBUG::2014-09-25 17:54:42,618::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71391::DEBUG::2014-09-25 17:54:42,618::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71391::DEBUG::2014-09-25 17:54:42,618::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71391::DEBUG::2014-09-25 17:54:42,619::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71391::DEBUG::2014-09-25 17:54:42,619::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71391::DEBUG::2014-09-25 17:54:42,619::task::993::Storage.TaskManager.Task::(_decref) Task=`c6ef6716-c3e2-497f-9545-ccb924eb1429`::ref 0 aborting False >Thread-71391::DEBUG::2014-09-25 17:54:42,620::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71391::DEBUG::2014-09-25 17:54:42,620::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:44,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:44,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144236 s, 71.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:44,863::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:44,865::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71392::DEBUG::2014-09-25 17:54:44,867::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef837692-8666-43d5-8af4-632a401adef9`::moving from state init -> state preparing >Thread-71392::INFO::2014-09-25 17:54:44,868::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71392::INFO::2014-09-25 17:54:44,868::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000425637', 'lastCheck': '3.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000422796', 'lastCheck': '7.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000390926', 'lastCheck': '9.7', 'valid': True}} >Thread-71392::DEBUG::2014-09-25 17:54:44,869::task::1191::Storage.TaskManager.Task::(prepare) Task=`ef837692-8666-43d5-8af4-632a401adef9`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000425637', 'lastCheck': '3.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000422796', 'lastCheck': '7.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000390926', 'lastCheck': '9.7', 'valid': True}} >Thread-71392::DEBUG::2014-09-25 17:54:44,869::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef837692-8666-43d5-8af4-632a401adef9`::moving from state preparing -> state finished >Thread-71392::DEBUG::2014-09-25 17:54:44,869::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71392::DEBUG::2014-09-25 17:54:44,869::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71392::DEBUG::2014-09-25 17:54:44,869::task::993::Storage.TaskManager.Task::(_decref) Task=`ef837692-8666-43d5-8af4-632a401adef9`::ref 0 aborting False >Thread-71392::DEBUG::2014-09-25 17:54:44,872::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:44,886::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:44,887::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71393::DEBUG::2014-09-25 17:54:44,889::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:54:45,196::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:45,196::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:45,197::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:45,197::__init__::232::IOProcess::(_processLogs) (109842) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:45,198::__init__::232::IOProcess::(_processLogs) (109842) Queuing response >Thread-81::DEBUG::2014-09-25 17:54:45,199::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:54:45,215::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00293024 s, 127 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:45,216::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:45,217::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:45,217::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:45,217::__init__::232::IOProcess::(_processLogs) (109843) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:45,219::__init__::232::IOProcess::(_processLogs) (109843) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:46,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:46,312::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0330184 s, 31.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:47,284::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:47,284::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:47,285::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:47,285::__init__::232::IOProcess::(_processLogs) (109844) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:47,285::__init__::232::IOProcess::(_processLogs) (109844) Queuing response >Thread-80::DEBUG::2014-09-25 17:54:47,286::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:54:47,294::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000351437 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:47,295::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:47,295::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:47,295::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:47,295::__init__::232::IOProcess::(_processLogs) (109845) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:47,296::__init__::232::IOProcess::(_processLogs) (109845) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:47,931::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:47,933::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71394::DEBUG::2014-09-25 17:54:47,935::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:48,510::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:48,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018029 s, 56.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:54:50,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:50,582::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136583 s, 75.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:50,935::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:50,936::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:50,936::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:50,937::__init__::232::IOProcess::(_processLogs) (109846) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:50,938::__init__::232::IOProcess::(_processLogs) (109846) Queuing response >Thread-79::DEBUG::2014-09-25 17:54:50,939::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:54:50,949::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000408594 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:50,950::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:50,950::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:50,951::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:50,951::__init__::232::IOProcess::(_processLogs) (109847) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:50,951::__init__::232::IOProcess::(_processLogs) (109847) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:50,953::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-29::DEBUG::2014-09-25 17:54:50,954::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:50,956::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >JsonRpcServer::DEBUG::2014-09-25 17:54:50,957::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-29::DEBUG::2014-09-25 17:54:50,959::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:50,960::__init__::232::IOProcess::(_processLogs) (109848) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:50,962::__init__::232::IOProcess::(_processLogs) (109848) Queuing response >Thread-71395::DEBUG::2014-09-25 17:54:50,963::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:54:50,964::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:50,966::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:50,967::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:50,967::__init__::232::IOProcess::(_processLogs) (109849) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:50,967::__init__::232::IOProcess::(_processLogs) (109849) Queuing response >Thread-29::DEBUG::2014-09-25 17:54:50,968::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:50,968::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:50,969::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:50,970::__init__::232::IOProcess::(_processLogs) (109850) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:54:50,970::__init__::232::IOProcess::(_processLogs) (109850) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:52,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:52,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136081 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:52,650::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:52,652::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71396::DEBUG::2014-09-25 17:54:52,652::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71396::DEBUG::2014-09-25 17:54:52,654::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f1c81a5-221f-4e10-8598-aa78186f771b`::moving from state init -> state preparing >Thread-71396::INFO::2014-09-25 17:54:52,655::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71396::INFO::2014-09-25 17:54:52,655::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71396::DEBUG::2014-09-25 17:54:52,655::task::1191::Storage.TaskManager.Task::(prepare) Task=`8f1c81a5-221f-4e10-8598-aa78186f771b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71396::DEBUG::2014-09-25 17:54:52,656::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f1c81a5-221f-4e10-8598-aa78186f771b`::moving from state preparing -> state finished >Thread-71396::DEBUG::2014-09-25 17:54:52,656::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71396::DEBUG::2014-09-25 17:54:52,656::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71396::DEBUG::2014-09-25 17:54:52,656::task::993::Storage.TaskManager.Task::(_decref) Task=`8f1c81a5-221f-4e10-8598-aa78186f771b`::ref 0 aborting False >Thread-71396::DEBUG::2014-09-25 17:54:52,657::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71396::DEBUG::2014-09-25 17:54:52,657::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:52,671::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71397::DEBUG::2014-09-25 17:54:52,672::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:54:52,672::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71397::DEBUG::2014-09-25 17:54:52,674::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3412835-5aa4-4a74-b203-53fb5f6ac553`::moving from state init -> state preparing >Thread-71397::INFO::2014-09-25 17:54:52,675::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71397::DEBUG::2014-09-25 17:54:52,675::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8eb4a60b-ce63-4295-8d6f-8c81b2679438`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71397::DEBUG::2014-09-25 17:54:52,676::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71397::DEBUG::2014-09-25 17:54:52,676::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71397::DEBUG::2014-09-25 17:54:52,676::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`8eb4a60b-ce63-4295-8d6f-8c81b2679438`::Granted request >Thread-71397::DEBUG::2014-09-25 17:54:52,677::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e3412835-5aa4-4a74-b203-53fb5f6ac553`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71397::DEBUG::2014-09-25 17:54:52,677::task::993::Storage.TaskManager.Task::(_decref) Task=`e3412835-5aa4-4a74-b203-53fb5f6ac553`::ref 1 aborting False >Thread-71397::INFO::2014-09-25 17:54:52,682::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71397::DEBUG::2014-09-25 17:54:52,682::task::1191::Storage.TaskManager.Task::(prepare) Task=`e3412835-5aa4-4a74-b203-53fb5f6ac553`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71397::DEBUG::2014-09-25 17:54:52,682::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3412835-5aa4-4a74-b203-53fb5f6ac553`::moving from state preparing -> state finished >Thread-71397::DEBUG::2014-09-25 17:54:52,682::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71397::DEBUG::2014-09-25 17:54:52,683::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71397::DEBUG::2014-09-25 17:54:52,683::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71397::DEBUG::2014-09-25 17:54:52,683::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71397::DEBUG::2014-09-25 17:54:52,684::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71397::DEBUG::2014-09-25 17:54:52,684::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71397::DEBUG::2014-09-25 17:54:52,684::task::993::Storage.TaskManager.Task::(_decref) Task=`e3412835-5aa4-4a74-b203-53fb5f6ac553`::ref 0 aborting False >Thread-71397::DEBUG::2014-09-25 17:54:52,685::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71397::DEBUG::2014-09-25 17:54:52,685::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:53,980::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:53,981::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71398::DEBUG::2014-09-25 17:54:53,983::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:54:54,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:54,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133426 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:55,221::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:55,221::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:55,222::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:55,222::__init__::232::IOProcess::(_processLogs) (109851) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:55,222::__init__::232::IOProcess::(_processLogs) (109851) Queuing response >Thread-81::DEBUG::2014-09-25 17:54:55,223::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:54:55,232::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000573396 s, 647 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:55,233::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:55,233::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:55,234::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:55,234::__init__::232::IOProcess::(_processLogs) (109852) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:55,234::__init__::232::IOProcess::(_processLogs) (109852) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:56,685::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:56,707::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133803 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:54:56,996::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:54:56,998::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71399::DEBUG::2014-09-25 17:54:57,000::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:54:57,297::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:57,297::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:57,298::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:57,298::__init__::232::IOProcess::(_processLogs) (109853) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:57,298::__init__::232::IOProcess::(_processLogs) (109853) Queuing response >Thread-80::DEBUG::2014-09-25 17:54:57,300::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:54:57,309::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000523906 s, 722 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:54:57,310::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:54:57,310::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:54:57,310::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:54:57,311::__init__::232::IOProcess::(_processLogs) (109854) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:54:57,311::__init__::232::IOProcess::(_processLogs) (109854) Queuing response >Dummy-148::DEBUG::2014-09-25 17:54:58,727::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:54:58,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132358 s, 77.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:00,014::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:00,016::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71400::DEBUG::2014-09-25 17:55:00,018::task::595::Storage.TaskManager.Task::(_updateState) Task=`2747d892-cbe8-4e40-b035-2be6f69c0128`::moving from state init -> state preparing >Thread-71400::INFO::2014-09-25 17:55:00,019::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71400::INFO::2014-09-25 17:55:00,019::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000408594', 'lastCheck': '9.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000523906', 'lastCheck': '2.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000573396', 'lastCheck': '4.8', 'valid': True}} >Thread-71400::DEBUG::2014-09-25 17:55:00,019::task::1191::Storage.TaskManager.Task::(prepare) Task=`2747d892-cbe8-4e40-b035-2be6f69c0128`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000408594', 'lastCheck': '9.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000523906', 'lastCheck': '2.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000573396', 'lastCheck': '4.8', 'valid': True}} >Thread-71400::DEBUG::2014-09-25 17:55:00,020::task::595::Storage.TaskManager.Task::(_updateState) Task=`2747d892-cbe8-4e40-b035-2be6f69c0128`::moving from state preparing -> state finished >Thread-71400::DEBUG::2014-09-25 17:55:00,020::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71400::DEBUG::2014-09-25 17:55:00,020::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71400::DEBUG::2014-09-25 17:55:00,020::task::993::Storage.TaskManager.Task::(_decref) Task=`2747d892-cbe8-4e40-b035-2be6f69c0128`::ref 0 aborting False >Thread-71400::DEBUG::2014-09-25 17:55:00,022::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:00,036::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:00,037::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71401::DEBUG::2014-09-25 17:55:00,039::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:00,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:00,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133573 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:00,971::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:00,971::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:00,971::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:00,971::__init__::232::IOProcess::(_processLogs) (109855) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:00,972::__init__::232::IOProcess::(_processLogs) (109855) Queuing response >Thread-79::DEBUG::2014-09-25 17:55:00,973::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:55:00,982::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000491108 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:00,983::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:00,983::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:00,984::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:00,984::__init__::232::IOProcess::(_processLogs) (109856) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:00,984::__init__::232::IOProcess::(_processLogs) (109856) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:00,986::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:00,986::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:00,987::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:00,987::__init__::232::IOProcess::(_processLogs) (109857) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:00,987::__init__::232::IOProcess::(_processLogs) (109857) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:00,988::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:00,988::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:00,989::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:00,989::__init__::232::IOProcess::(_processLogs) (109858) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:00,990::__init__::232::IOProcess::(_processLogs) (109858) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:00,990::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:00,990::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:00,991::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:00,992::__init__::232::IOProcess::(_processLogs) (109859) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:00,992::__init__::232::IOProcess::(_processLogs) (109859) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:02,715::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:02,716::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71402::DEBUG::2014-09-25 17:55:02,716::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71402::DEBUG::2014-09-25 17:55:02,718::task::595::Storage.TaskManager.Task::(_updateState) Task=`e4f929bc-314d-4244-8d6a-ce7709b81d3f`::moving from state init -> state preparing >Thread-71402::INFO::2014-09-25 17:55:02,719::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71402::INFO::2014-09-25 17:55:02,719::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71402::DEBUG::2014-09-25 17:55:02,719::task::1191::Storage.TaskManager.Task::(prepare) Task=`e4f929bc-314d-4244-8d6a-ce7709b81d3f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71402::DEBUG::2014-09-25 17:55:02,720::task::595::Storage.TaskManager.Task::(_updateState) Task=`e4f929bc-314d-4244-8d6a-ce7709b81d3f`::moving from state preparing -> state finished >Thread-71402::DEBUG::2014-09-25 17:55:02,720::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71402::DEBUG::2014-09-25 17:55:02,720::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71402::DEBUG::2014-09-25 17:55:02,720::task::993::Storage.TaskManager.Task::(_decref) Task=`e4f929bc-314d-4244-8d6a-ce7709b81d3f`::ref 0 aborting False >Thread-71402::DEBUG::2014-09-25 17:55:02,721::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71402::DEBUG::2014-09-25 17:55:02,721::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:02,731::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71403::DEBUG::2014-09-25 17:55:02,732::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:55:02,732::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71403::DEBUG::2014-09-25 17:55:02,734::task::595::Storage.TaskManager.Task::(_updateState) Task=`9b27d550-48ad-46c5-a5aa-f801e11369cf`::moving from state init -> state preparing >Thread-71403::INFO::2014-09-25 17:55:02,735::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71403::DEBUG::2014-09-25 17:55:02,735::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ab816012-59a4-4db2-a484-e27b309ff1ac`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71403::DEBUG::2014-09-25 17:55:02,736::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71403::DEBUG::2014-09-25 17:55:02,736::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71403::DEBUG::2014-09-25 17:55:02,737::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ab816012-59a4-4db2-a484-e27b309ff1ac`::Granted request >Thread-71403::DEBUG::2014-09-25 17:55:02,737::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9b27d550-48ad-46c5-a5aa-f801e11369cf`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71403::DEBUG::2014-09-25 17:55:02,737::task::993::Storage.TaskManager.Task::(_decref) Task=`9b27d550-48ad-46c5-a5aa-f801e11369cf`::ref 1 aborting False >Thread-71403::INFO::2014-09-25 17:55:02,742::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71403::DEBUG::2014-09-25 17:55:02,742::task::1191::Storage.TaskManager.Task::(prepare) Task=`9b27d550-48ad-46c5-a5aa-f801e11369cf`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71403::DEBUG::2014-09-25 17:55:02,742::task::595::Storage.TaskManager.Task::(_updateState) Task=`9b27d550-48ad-46c5-a5aa-f801e11369cf`::moving from state preparing -> state finished >Thread-71403::DEBUG::2014-09-25 17:55:02,743::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71403::DEBUG::2014-09-25 17:55:02,743::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71403::DEBUG::2014-09-25 17:55:02,743::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71403::DEBUG::2014-09-25 17:55:02,744::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71403::DEBUG::2014-09-25 17:55:02,744::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71403::DEBUG::2014-09-25 17:55:02,744::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71403::DEBUG::2014-09-25 17:55:02,744::task::993::Storage.TaskManager.Task::(_decref) Task=`9b27d550-48ad-46c5-a5aa-f801e11369cf`::ref 0 aborting False >Thread-71403::DEBUG::2014-09-25 17:55:02,745::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71403::DEBUG::2014-09-25 17:55:02,745::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:02,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:02,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134418 s, 76.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:03,083::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:03,085::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71404::DEBUG::2014-09-25 17:55:03,087::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:04,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:04,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133914 s, 76.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:05,236::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:05,236::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:05,237::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:05,237::__init__::232::IOProcess::(_processLogs) (109860) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:05,237::__init__::232::IOProcess::(_processLogs) (109860) Queuing response >Thread-81::DEBUG::2014-09-25 17:55:05,238::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:55:05,247::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000476585 s, 778 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:05,248::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:05,248::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:05,249::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:05,250::__init__::232::IOProcess::(_processLogs) (109861) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:05,251::__init__::232::IOProcess::(_processLogs) (109861) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:06,099::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:06,100::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71405::DEBUG::2014-09-25 17:55:06,103::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:06,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:06,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.054763 s, 18.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:07,314::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:07,314::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:07,314::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:07,314::__init__::232::IOProcess::(_processLogs) (109862) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:07,315::__init__::232::IOProcess::(_processLogs) (109862) Queuing response >Thread-80::DEBUG::2014-09-25 17:55:07,316::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:55:07,325::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000552495 s, 684 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:07,327::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:07,328::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:07,329::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:07,329::__init__::232::IOProcess::(_processLogs) (109863) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:07,329::__init__::232::IOProcess::(_processLogs) (109863) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:08,979::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:09,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134414 s, 76.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:09,115::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:09,116::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71406::DEBUG::2014-09-25 17:55:09,119::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:55:10,992::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:10,993::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:10,993::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:10,993::__init__::232::IOProcess::(_processLogs) (109864) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:10,994::__init__::232::IOProcess::(_processLogs) (109864) Queuing response >Thread-79::DEBUG::2014-09-25 17:55:10,995::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:55:11,004::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000516454 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:11,005::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:11,006::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:11,006::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:11,006::__init__::232::IOProcess::(_processLogs) (109865) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:11,007::__init__::232::IOProcess::(_processLogs) (109865) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:11,008::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:11,009::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:11,009::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:11,010::__init__::232::IOProcess::(_processLogs) (109866) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:11,010::__init__::232::IOProcess::(_processLogs) (109866) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:11,010::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:11,011::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:11,012::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:11,012::__init__::232::IOProcess::(_processLogs) (109867) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:11,012::__init__::232::IOProcess::(_processLogs) (109867) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:11,013::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:11,013::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:11,013::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:11,014::__init__::232::IOProcess::(_processLogs) (109868) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:11,014::__init__::232::IOProcess::(_processLogs) (109868) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:11,021::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:11,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132022 s, 77.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:12,134::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:12,136::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71407::DEBUG::2014-09-25 17:55:12,138::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:12,769::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71408::DEBUG::2014-09-25 17:55:12,770::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:55:12,771::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71408::DEBUG::2014-09-25 17:55:12,773::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2ffb578-f0a9-4f5c-9dd1-38dd8fd88025`::moving from state init -> state preparing >Thread-71408::INFO::2014-09-25 17:55:12,773::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71408::INFO::2014-09-25 17:55:12,774::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71408::DEBUG::2014-09-25 17:55:12,774::task::1191::Storage.TaskManager.Task::(prepare) Task=`d2ffb578-f0a9-4f5c-9dd1-38dd8fd88025`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71408::DEBUG::2014-09-25 17:55:12,774::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2ffb578-f0a9-4f5c-9dd1-38dd8fd88025`::moving from state preparing -> state finished >Thread-71408::DEBUG::2014-09-25 17:55:12,775::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71408::DEBUG::2014-09-25 17:55:12,775::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71408::DEBUG::2014-09-25 17:55:12,775::task::993::Storage.TaskManager.Task::(_decref) Task=`d2ffb578-f0a9-4f5c-9dd1-38dd8fd88025`::ref 0 aborting False >Thread-71408::DEBUG::2014-09-25 17:55:12,775::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71408::DEBUG::2014-09-25 17:55:12,776::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:12,791::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71409::DEBUG::2014-09-25 17:55:12,792::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:55:12,792::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71409::DEBUG::2014-09-25 17:55:12,794::task::595::Storage.TaskManager.Task::(_updateState) Task=`69b14579-3b93-482e-8139-d61ae91dd197`::moving from state init -> state preparing >Thread-71409::INFO::2014-09-25 17:55:12,794::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71409::DEBUG::2014-09-25 17:55:12,795::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`599e47ca-b3a4-4b49-85ae-af0e4ef17c36`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71409::DEBUG::2014-09-25 17:55:12,795::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71409::DEBUG::2014-09-25 17:55:12,796::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71409::DEBUG::2014-09-25 17:55:12,796::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`599e47ca-b3a4-4b49-85ae-af0e4ef17c36`::Granted request >Thread-71409::DEBUG::2014-09-25 17:55:12,797::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`69b14579-3b93-482e-8139-d61ae91dd197`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71409::DEBUG::2014-09-25 17:55:12,797::task::993::Storage.TaskManager.Task::(_decref) Task=`69b14579-3b93-482e-8139-d61ae91dd197`::ref 1 aborting False >Thread-71409::INFO::2014-09-25 17:55:12,801::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71409::DEBUG::2014-09-25 17:55:12,802::task::1191::Storage.TaskManager.Task::(prepare) Task=`69b14579-3b93-482e-8139-d61ae91dd197`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71409::DEBUG::2014-09-25 17:55:12,802::task::595::Storage.TaskManager.Task::(_updateState) Task=`69b14579-3b93-482e-8139-d61ae91dd197`::moving from state preparing -> state finished >Thread-71409::DEBUG::2014-09-25 17:55:12,802::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71409::DEBUG::2014-09-25 17:55:12,802::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71409::DEBUG::2014-09-25 17:55:12,803::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71409::DEBUG::2014-09-25 17:55:12,803::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71409::DEBUG::2014-09-25 17:55:12,803::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71409::DEBUG::2014-09-25 17:55:12,803::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71409::DEBUG::2014-09-25 17:55:12,804::task::993::Storage.TaskManager.Task::(_decref) Task=`69b14579-3b93-482e-8139-d61ae91dd197`::ref 0 aborting False >Thread-71409::DEBUG::2014-09-25 17:55:12,804::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71409::DEBUG::2014-09-25 17:55:12,805::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:13,063::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:13,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134745 s, 76.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:55:15,106::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:15,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133345 s, 76.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:15,154::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:15,156::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71410::DEBUG::2014-09-25 17:55:15,159::task::595::Storage.TaskManager.Task::(_updateState) Task=`a288c6e1-a90a-49da-969b-44f673f40937`::moving from state init -> state preparing >Thread-71410::INFO::2014-09-25 17:55:15,159::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71410::INFO::2014-09-25 17:55:15,160::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000516454', 'lastCheck': '4.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000552495', 'lastCheck': '7.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000476585', 'lastCheck': '9.9', 'valid': True}} >Thread-71410::DEBUG::2014-09-25 17:55:15,160::task::1191::Storage.TaskManager.Task::(prepare) Task=`a288c6e1-a90a-49da-969b-44f673f40937`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000516454', 'lastCheck': '4.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000552495', 'lastCheck': '7.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000476585', 'lastCheck': '9.9', 'valid': True}} >Thread-71410::DEBUG::2014-09-25 17:55:15,160::task::595::Storage.TaskManager.Task::(_updateState) Task=`a288c6e1-a90a-49da-969b-44f673f40937`::moving from state preparing -> state finished >Thread-71410::DEBUG::2014-09-25 17:55:15,161::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71410::DEBUG::2014-09-25 17:55:15,161::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71410::DEBUG::2014-09-25 17:55:15,161::task::993::Storage.TaskManager.Task::(_decref) Task=`a288c6e1-a90a-49da-969b-44f673f40937`::ref 0 aborting False >Thread-71410::DEBUG::2014-09-25 17:55:15,164::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:15,188::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:15,190::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71411::DEBUG::2014-09-25 17:55:15,191::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:55:15,251::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:15,251::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:15,251::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:15,252::__init__::232::IOProcess::(_processLogs) (109869) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:15,253::__init__::232::IOProcess::(_processLogs) (109869) Queuing response >Thread-81::DEBUG::2014-09-25 17:55:15,253::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:55:15,262::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000468036 s, 793 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:15,263::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:15,264::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:15,264::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:15,266::__init__::232::IOProcess::(_processLogs) (109870) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:15,266::__init__::232::IOProcess::(_processLogs) (109870) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:17,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:17,172::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159608 s, 64.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:17,329::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:17,330::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:17,330::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:17,330::__init__::232::IOProcess::(_processLogs) (109871) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:17,330::__init__::232::IOProcess::(_processLogs) (109871) Queuing response >Thread-80::DEBUG::2014-09-25 17:55:17,331::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:55:17,338::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00047251 s, 800 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:17,339::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:17,339::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:17,339::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:17,339::__init__::232::IOProcess::(_processLogs) (109872) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:17,340::__init__::232::IOProcess::(_processLogs) (109872) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:18,256::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:18,258::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71412::DEBUG::2014-09-25 17:55:18,260::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:19,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:19,215::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015959 s, 64.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:21,015::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:21,015::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:21,016::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:21,016::__init__::232::IOProcess::(_processLogs) (109873) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:21,016::__init__::232::IOProcess::(_processLogs) (109873) Queuing response >Thread-79::DEBUG::2014-09-25 17:55:21,018::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:55:21,026::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000446866 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:21,028::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:21,028::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:21,029::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:21,030::__init__::232::IOProcess::(_processLogs) (109874) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:21,030::__init__::232::IOProcess::(_processLogs) (109874) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:21,031::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:21,031::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:21,031::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:21,031::__init__::232::IOProcess::(_processLogs) (109875) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:21,032::__init__::232::IOProcess::(_processLogs) (109875) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:21,032::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:21,033::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:21,033::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:21,034::__init__::232::IOProcess::(_processLogs) (109876) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:21,034::__init__::232::IOProcess::(_processLogs) (109876) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:21,035::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:21,035::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:21,035::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:21,035::__init__::232::IOProcess::(_processLogs) (109877) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:21,036::__init__::232::IOProcess::(_processLogs) (109877) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:21,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:21,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133957 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:21,275::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:21,278::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71413::DEBUG::2014-09-25 17:55:21,282::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:22,834::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:22,836::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71414::DEBUG::2014-09-25 17:55:22,836::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71414::DEBUG::2014-09-25 17:55:22,838::task::595::Storage.TaskManager.Task::(_updateState) Task=`7f2171d3-6b71-438e-9d94-1eb96f49f7ae`::moving from state init -> state preparing >Thread-71414::INFO::2014-09-25 17:55:22,838::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71414::INFO::2014-09-25 17:55:22,839::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71414::DEBUG::2014-09-25 17:55:22,839::task::1191::Storage.TaskManager.Task::(prepare) Task=`7f2171d3-6b71-438e-9d94-1eb96f49f7ae`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71414::DEBUG::2014-09-25 17:55:22,839::task::595::Storage.TaskManager.Task::(_updateState) Task=`7f2171d3-6b71-438e-9d94-1eb96f49f7ae`::moving from state preparing -> state finished >Thread-71414::DEBUG::2014-09-25 17:55:22,839::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71414::DEBUG::2014-09-25 17:55:22,840::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71414::DEBUG::2014-09-25 17:55:22,840::task::993::Storage.TaskManager.Task::(_decref) Task=`7f2171d3-6b71-438e-9d94-1eb96f49f7ae`::ref 0 aborting False >Thread-71414::DEBUG::2014-09-25 17:55:22,840::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71414::DEBUG::2014-09-25 17:55:22,841::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:22,851::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71415::DEBUG::2014-09-25 17:55:22,852::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:55:22,852::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71415::DEBUG::2014-09-25 17:55:22,855::task::595::Storage.TaskManager.Task::(_updateState) Task=`efd280bc-c1b5-48c0-a2be-b21b455802ef`::moving from state init -> state preparing >Thread-71415::INFO::2014-09-25 17:55:22,855::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71415::DEBUG::2014-09-25 17:55:22,856::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`27a9137d-1945-4882-9d36-54f7f7e979e7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71415::DEBUG::2014-09-25 17:55:22,856::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71415::DEBUG::2014-09-25 17:55:22,856::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71415::DEBUG::2014-09-25 17:55:22,856::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`27a9137d-1945-4882-9d36-54f7f7e979e7`::Granted request >Thread-71415::DEBUG::2014-09-25 17:55:22,857::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`efd280bc-c1b5-48c0-a2be-b21b455802ef`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71415::DEBUG::2014-09-25 17:55:22,857::task::993::Storage.TaskManager.Task::(_decref) Task=`efd280bc-c1b5-48c0-a2be-b21b455802ef`::ref 1 aborting False >Thread-71415::INFO::2014-09-25 17:55:22,862::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71415::DEBUG::2014-09-25 17:55:22,862::task::1191::Storage.TaskManager.Task::(prepare) Task=`efd280bc-c1b5-48c0-a2be-b21b455802ef`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71415::DEBUG::2014-09-25 17:55:22,862::task::595::Storage.TaskManager.Task::(_updateState) Task=`efd280bc-c1b5-48c0-a2be-b21b455802ef`::moving from state preparing -> state finished >Thread-71415::DEBUG::2014-09-25 17:55:22,862::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71415::DEBUG::2014-09-25 17:55:22,863::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71415::DEBUG::2014-09-25 17:55:22,863::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71415::DEBUG::2014-09-25 17:55:22,863::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71415::DEBUG::2014-09-25 17:55:22,863::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71415::DEBUG::2014-09-25 17:55:22,864::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71415::DEBUG::2014-09-25 17:55:22,864::task::993::Storage.TaskManager.Task::(_decref) Task=`efd280bc-c1b5-48c0-a2be-b21b455802ef`::ref 0 aborting False >Thread-71415::DEBUG::2014-09-25 17:55:22,864::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7250903040', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71415::DEBUG::2014-09-25 17:55:22,865::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:23,286::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:23,309::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0139832 s, 73.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:24,297::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:24,298::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71416::DEBUG::2014-09-25 17:55:24,300::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:55:25,267::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:25,267::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:25,268::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:25,269::__init__::232::IOProcess::(_processLogs) (109878) Got request for method 'statvfs' >Thread-81::DEBUG::2014-09-25 17:55:25,269::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:55:25,278::__init__::232::IOProcess::(_processLogs) (109878) Queuing response >Thread-81::DEBUG::2014-09-25 17:55:25,279::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000653808 s, 567 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:25,280::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:25,281::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:25,282::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:25,283::__init__::232::IOProcess::(_processLogs) (109879) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:25,283::__init__::232::IOProcess::(_processLogs) (109879) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:25,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:25,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166386 s, 61.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:27,314::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:27,316::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71417::DEBUG::2014-09-25 17:55:27,318::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:55:27,341::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:27,342::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:27,342::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:27,342::__init__::232::IOProcess::(_processLogs) (109880) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:27,343::__init__::232::IOProcess::(_processLogs) (109880) Queuing response >Thread-80::DEBUG::2014-09-25 17:55:27,344::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:55:27,353::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000378364 s, 999 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:27,354::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:27,355::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:27,355::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:27,355::__init__::232::IOProcess::(_processLogs) (109881) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:27,356::__init__::232::IOProcess::(_processLogs) (109881) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:27,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:27,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132218 s, 77.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:55:29,415::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:29,437::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131533 s, 77.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:30,332::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:30,333::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71418::DEBUG::2014-09-25 17:55:30,336::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f0c9ea4-e3b2-4ada-b10b-872e7ac5d029`::moving from state init -> state preparing >Thread-71418::INFO::2014-09-25 17:55:30,336::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71418::INFO::2014-09-25 17:55:30,337::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000446866', 'lastCheck': '9.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000378364', 'lastCheck': '3.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000653808', 'lastCheck': '5.1', 'valid': True}} >Thread-71418::DEBUG::2014-09-25 17:55:30,337::task::1191::Storage.TaskManager.Task::(prepare) Task=`8f0c9ea4-e3b2-4ada-b10b-872e7ac5d029`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000446866', 'lastCheck': '9.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000378364', 'lastCheck': '3.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000653808', 'lastCheck': '5.1', 'valid': True}} >Thread-71418::DEBUG::2014-09-25 17:55:30,337::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f0c9ea4-e3b2-4ada-b10b-872e7ac5d029`::moving from state preparing -> state finished >Thread-71418::DEBUG::2014-09-25 17:55:30,337::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71418::DEBUG::2014-09-25 17:55:30,338::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71418::DEBUG::2014-09-25 17:55:30,338::task::993::Storage.TaskManager.Task::(_decref) Task=`8f0c9ea4-e3b2-4ada-b10b-872e7ac5d029`::ref 0 aborting False >Thread-71418::DEBUG::2014-09-25 17:55:30,340::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:30,358::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:30,359::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71419::DEBUG::2014-09-25 17:55:30,361::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:55:31,037::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:31,037::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:31,038::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:31,038::__init__::232::IOProcess::(_processLogs) (109882) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:31,038::__init__::232::IOProcess::(_processLogs) (109882) Queuing response >Thread-79::DEBUG::2014-09-25 17:55:31,040::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:55:31,049::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00039677 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:31,050::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:31,050::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:31,050::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:31,051::__init__::232::IOProcess::(_processLogs) (109883) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:31,051::__init__::232::IOProcess::(_processLogs) (109883) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:31,053::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:31,053::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:31,053::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:31,053::__init__::232::IOProcess::(_processLogs) (109884) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:31,054::__init__::232::IOProcess::(_processLogs) (109884) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:31,054::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:31,055::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:31,055::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:31,055::__init__::232::IOProcess::(_processLogs) (109885) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:31,056::__init__::232::IOProcess::(_processLogs) (109885) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:31,057::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:31,057::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:31,057::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:31,057::__init__::232::IOProcess::(_processLogs) (109886) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:31,058::__init__::232::IOProcess::(_processLogs) (109886) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:31,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:31,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132967 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:32,894::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71420::DEBUG::2014-09-25 17:55:32,895::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:55:32,895::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71420::DEBUG::2014-09-25 17:55:32,897::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f6a1036-a1bd-4112-aed4-7953dfce0dca`::moving from state init -> state preparing >Thread-71420::INFO::2014-09-25 17:55:32,898::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71420::INFO::2014-09-25 17:55:32,898::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71420::DEBUG::2014-09-25 17:55:32,898::task::1191::Storage.TaskManager.Task::(prepare) Task=`9f6a1036-a1bd-4112-aed4-7953dfce0dca`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71420::DEBUG::2014-09-25 17:55:32,899::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f6a1036-a1bd-4112-aed4-7953dfce0dca`::moving from state preparing -> state finished >Thread-71420::DEBUG::2014-09-25 17:55:32,899::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71420::DEBUG::2014-09-25 17:55:32,899::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71420::DEBUG::2014-09-25 17:55:32,899::task::993::Storage.TaskManager.Task::(_decref) Task=`9f6a1036-a1bd-4112-aed4-7953dfce0dca`::ref 0 aborting False >Thread-71420::DEBUG::2014-09-25 17:55:32,900::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71420::DEBUG::2014-09-25 17:55:32,900::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:32,911::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:32,912::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71421::DEBUG::2014-09-25 17:55:32,912::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71421::DEBUG::2014-09-25 17:55:32,914::task::595::Storage.TaskManager.Task::(_updateState) Task=`8abf20e8-8635-43e2-9ea3-feadf9a33e1f`::moving from state init -> state preparing >Thread-71421::INFO::2014-09-25 17:55:32,914::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71421::DEBUG::2014-09-25 17:55:32,915::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`55632bca-35d8-412f-8344-83bc61ae4c3b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71421::DEBUG::2014-09-25 17:55:32,915::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71421::DEBUG::2014-09-25 17:55:32,915::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71421::DEBUG::2014-09-25 17:55:32,916::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`55632bca-35d8-412f-8344-83bc61ae4c3b`::Granted request >Thread-71421::DEBUG::2014-09-25 17:55:32,916::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`8abf20e8-8635-43e2-9ea3-feadf9a33e1f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71421::DEBUG::2014-09-25 17:55:32,916::task::993::Storage.TaskManager.Task::(_decref) Task=`8abf20e8-8635-43e2-9ea3-feadf9a33e1f`::ref 1 aborting False >Thread-71421::INFO::2014-09-25 17:55:32,921::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7243563008', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71421::DEBUG::2014-09-25 17:55:32,921::task::1191::Storage.TaskManager.Task::(prepare) Task=`8abf20e8-8635-43e2-9ea3-feadf9a33e1f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7243563008', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71421::DEBUG::2014-09-25 17:55:32,921::task::595::Storage.TaskManager.Task::(_updateState) Task=`8abf20e8-8635-43e2-9ea3-feadf9a33e1f`::moving from state preparing -> state finished >Thread-71421::DEBUG::2014-09-25 17:55:32,922::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71421::DEBUG::2014-09-25 17:55:32,922::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71421::DEBUG::2014-09-25 17:55:32,922::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71421::DEBUG::2014-09-25 17:55:32,922::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71421::DEBUG::2014-09-25 17:55:32,923::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71421::DEBUG::2014-09-25 17:55:32,923::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71421::DEBUG::2014-09-25 17:55:32,923::task::993::Storage.TaskManager.Task::(_decref) Task=`8abf20e8-8635-43e2-9ea3-feadf9a33e1f`::ref 0 aborting False >Thread-71421::DEBUG::2014-09-25 17:55:32,924::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7243563008', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71421::DEBUG::2014-09-25 17:55:32,924::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:33,429::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:33,430::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71422::DEBUG::2014-09-25 17:55:33,432::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:33,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:33,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136162 s, 75.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:35,284::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:35,284::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:35,284::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:35,285::__init__::232::IOProcess::(_processLogs) (109887) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:35,285::__init__::232::IOProcess::(_processLogs) (109887) Queuing response >Thread-81::DEBUG::2014-09-25 17:55:35,286::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:55:35,295::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000528577 s, 702 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:35,296::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:35,297::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:35,297::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:35,297::__init__::232::IOProcess::(_processLogs) (109888) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:35,298::__init__::232::IOProcess::(_processLogs) (109888) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:35,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:35,560::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132702 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:36,447::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:36,449::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71423::DEBUG::2014-09-25 17:55:36,451::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:55:37,358::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:37,359::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:37,359::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:55:37,360::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:55:37,360::__init__::232::IOProcess::(_processLogs) (109889) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:37,369::__init__::232::IOProcess::(_processLogs) (109889) Queuing response >Thread-80::DEBUG::2014-09-25 17:55:37,370::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000337652 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:37,371::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:37,371::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:37,371::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:37,372::__init__::232::IOProcess::(_processLogs) (109890) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:37,372::__init__::232::IOProcess::(_processLogs) (109890) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:37,580::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:37,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016438 s, 62.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:39,464::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:39,465::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71424::DEBUG::2014-09-25 17:55:39,467::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:39,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:39,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183448 s, 55.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:41,059::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:41,060::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:41,060::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:41,060::__init__::232::IOProcess::(_processLogs) (109891) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:41,061::__init__::232::IOProcess::(_processLogs) (109891) Queuing response >Thread-79::DEBUG::2014-09-25 17:55:41,062::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:55:41,071::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000492632 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:41,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:41,072::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:41,073::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:41,073::__init__::232::IOProcess::(_processLogs) (109892) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:41,074::__init__::232::IOProcess::(_processLogs) (109892) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:41,075::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:41,075::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:41,075::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:41,076::__init__::232::IOProcess::(_processLogs) (109893) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:41,076::__init__::232::IOProcess::(_processLogs) (109893) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:41,077::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:41,077::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:41,077::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:41,078::__init__::232::IOProcess::(_processLogs) (109894) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:41,078::__init__::232::IOProcess::(_processLogs) (109894) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:41,079::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:41,079::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:41,079::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:41,080::__init__::232::IOProcess::(_processLogs) (109895) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:41,080::__init__::232::IOProcess::(_processLogs) (109895) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:41,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:41,704::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138908 s, 73.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:42,479::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:42,481::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71425::DEBUG::2014-09-25 17:55:42,483::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:42,949::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71426::DEBUG::2014-09-25 17:55:42,951::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:55:42,951::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71426::DEBUG::2014-09-25 17:55:42,953::task::595::Storage.TaskManager.Task::(_updateState) Task=`79ce7655-a401-43dc-9001-4a588a848f93`::moving from state init -> state preparing >Thread-71426::INFO::2014-09-25 17:55:42,954::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71426::INFO::2014-09-25 17:55:42,954::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71426::DEBUG::2014-09-25 17:55:42,955::task::1191::Storage.TaskManager.Task::(prepare) Task=`79ce7655-a401-43dc-9001-4a588a848f93`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71426::DEBUG::2014-09-25 17:55:42,955::task::595::Storage.TaskManager.Task::(_updateState) Task=`79ce7655-a401-43dc-9001-4a588a848f93`::moving from state preparing -> state finished >Thread-71426::DEBUG::2014-09-25 17:55:42,955::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71426::DEBUG::2014-09-25 17:55:42,955::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71426::DEBUG::2014-09-25 17:55:42,955::task::993::Storage.TaskManager.Task::(_decref) Task=`79ce7655-a401-43dc-9001-4a588a848f93`::ref 0 aborting False >Thread-71426::DEBUG::2014-09-25 17:55:42,956::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71426::DEBUG::2014-09-25 17:55:42,956::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:42,966::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71427::DEBUG::2014-09-25 17:55:42,967::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:55:42,968::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71427::DEBUG::2014-09-25 17:55:42,970::task::595::Storage.TaskManager.Task::(_updateState) Task=`42342470-3e81-408e-b719-5a9a0e8f68d9`::moving from state init -> state preparing >Thread-71427::INFO::2014-09-25 17:55:42,970::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71427::DEBUG::2014-09-25 17:55:42,971::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f6e5013a-e729-4157-8df7-dfa541c2bc05`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71427::DEBUG::2014-09-25 17:55:42,971::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71427::DEBUG::2014-09-25 17:55:42,971::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71427::DEBUG::2014-09-25 17:55:42,972::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f6e5013a-e729-4157-8df7-dfa541c2bc05`::Granted request >Thread-71427::DEBUG::2014-09-25 17:55:42,972::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`42342470-3e81-408e-b719-5a9a0e8f68d9`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71427::DEBUG::2014-09-25 17:55:42,972::task::993::Storage.TaskManager.Task::(_decref) Task=`42342470-3e81-408e-b719-5a9a0e8f68d9`::ref 1 aborting False >Thread-71427::INFO::2014-09-25 17:55:42,977::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71427::DEBUG::2014-09-25 17:55:42,977::task::1191::Storage.TaskManager.Task::(prepare) Task=`42342470-3e81-408e-b719-5a9a0e8f68d9`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71427::DEBUG::2014-09-25 17:55:42,977::task::595::Storage.TaskManager.Task::(_updateState) Task=`42342470-3e81-408e-b719-5a9a0e8f68d9`::moving from state preparing -> state finished >Thread-71427::DEBUG::2014-09-25 17:55:42,978::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71427::DEBUG::2014-09-25 17:55:42,978::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71427::DEBUG::2014-09-25 17:55:42,978::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71427::DEBUG::2014-09-25 17:55:42,978::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71427::DEBUG::2014-09-25 17:55:42,979::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71427::DEBUG::2014-09-25 17:55:42,979::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71427::DEBUG::2014-09-25 17:55:42,979::task::993::Storage.TaskManager.Task::(_decref) Task=`42342470-3e81-408e-b719-5a9a0e8f68d9`::ref 0 aborting False >Thread-71427::DEBUG::2014-09-25 17:55:42,980::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71427::DEBUG::2014-09-25 17:55:42,980::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:43,724::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:43,753::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018674 s, 54.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:45,300::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:45,301::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:45,301::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:55:45,302::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:55:45,303::__init__::232::IOProcess::(_processLogs) (109896) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:45,311::__init__::232::IOProcess::(_processLogs) (109896) Queuing response >Thread-81::DEBUG::2014-09-25 17:55:45,312::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000427772 s, 867 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:45,313::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:45,314::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:45,314::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:45,315::__init__::232::IOProcess::(_processLogs) (109897) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:45,316::__init__::232::IOProcess::(_processLogs) (109897) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:45,495::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:45,496::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71428::DEBUG::2014-09-25 17:55:45,499::task::595::Storage.TaskManager.Task::(_updateState) Task=`ca2a2fde-2323-47a9-82ff-04d6bb3778b3`::moving from state init -> state preparing >Thread-71428::INFO::2014-09-25 17:55:45,499::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71428::INFO::2014-09-25 17:55:45,500::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000492632', 'lastCheck': '4.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000337652', 'lastCheck': '8.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000427772', 'lastCheck': '0.2', 'valid': True}} >Thread-71428::DEBUG::2014-09-25 17:55:45,500::task::1191::Storage.TaskManager.Task::(prepare) Task=`ca2a2fde-2323-47a9-82ff-04d6bb3778b3`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000492632', 'lastCheck': '4.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000337652', 'lastCheck': '8.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000427772', 'lastCheck': '0.2', 'valid': True}} >Thread-71428::DEBUG::2014-09-25 17:55:45,500::task::595::Storage.TaskManager.Task::(_updateState) Task=`ca2a2fde-2323-47a9-82ff-04d6bb3778b3`::moving from state preparing -> state finished >Thread-71428::DEBUG::2014-09-25 17:55:45,501::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71428::DEBUG::2014-09-25 17:55:45,501::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71428::DEBUG::2014-09-25 17:55:45,501::task::993::Storage.TaskManager.Task::(_decref) Task=`ca2a2fde-2323-47a9-82ff-04d6bb3778b3`::ref 0 aborting False >Thread-71428::DEBUG::2014-09-25 17:55:45,504::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:45,523::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:45,524::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71429::DEBUG::2014-09-25 17:55:45,526::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:45,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:45,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013384 s, 76.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:47,374::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:47,374::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:47,375::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:47,375::__init__::232::IOProcess::(_processLogs) (109898) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:47,375::__init__::232::IOProcess::(_processLogs) (109898) Queuing response >Thread-80::DEBUG::2014-09-25 17:55:47,376::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:55:47,383::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000369876 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:47,384::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:47,384::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:47,384::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:47,385::__init__::232::IOProcess::(_processLogs) (109899) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:47,385::__init__::232::IOProcess::(_processLogs) (109899) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:47,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:47,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137594 s, 74.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:48,572::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:48,573::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71430::DEBUG::2014-09-25 17:55:48,575::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:49,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:49,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013276 s, 77.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:51,082::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:51,082::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:51,083::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:51,083::__init__::232::IOProcess::(_processLogs) (109900) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:51,083::__init__::232::IOProcess::(_processLogs) (109900) Queuing response >Thread-79::DEBUG::2014-09-25 17:55:51,084::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:55:51,093::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000484509 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:51,094::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:51,095::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:51,095::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:51,095::__init__::232::IOProcess::(_processLogs) (109901) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:51,096::__init__::232::IOProcess::(_processLogs) (109901) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:51,097::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:51,097::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:51,098::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:51,098::__init__::232::IOProcess::(_processLogs) (109902) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:51,099::__init__::232::IOProcess::(_processLogs) (109902) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:51,099::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:51,100::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:51,100::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:51,100::__init__::232::IOProcess::(_processLogs) (109903) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:51,101::__init__::232::IOProcess::(_processLogs) (109903) Queuing response >Thread-29::DEBUG::2014-09-25 17:55:51,102::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:51,102::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:51,102::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:51,102::__init__::232::IOProcess::(_processLogs) (109904) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:55:51,103::__init__::232::IOProcess::(_processLogs) (109904) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:51,585::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:51,587::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71431::DEBUG::2014-09-25 17:55:51,589::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:51,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:51,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136191 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:53,007::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:53,008::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71432::DEBUG::2014-09-25 17:55:53,009::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71432::DEBUG::2014-09-25 17:55:53,011::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5c00812-1d15-48fc-bad1-e860ece5cbb2`::moving from state init -> state preparing >Thread-71432::INFO::2014-09-25 17:55:53,011::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71432::INFO::2014-09-25 17:55:53,012::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71432::DEBUG::2014-09-25 17:55:53,012::task::1191::Storage.TaskManager.Task::(prepare) Task=`f5c00812-1d15-48fc-bad1-e860ece5cbb2`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71432::DEBUG::2014-09-25 17:55:53,012::task::595::Storage.TaskManager.Task::(_updateState) Task=`f5c00812-1d15-48fc-bad1-e860ece5cbb2`::moving from state preparing -> state finished >Thread-71432::DEBUG::2014-09-25 17:55:53,013::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71432::DEBUG::2014-09-25 17:55:53,013::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71432::DEBUG::2014-09-25 17:55:53,013::task::993::Storage.TaskManager.Task::(_decref) Task=`f5c00812-1d15-48fc-bad1-e860ece5cbb2`::ref 0 aborting False >Thread-71432::DEBUG::2014-09-25 17:55:53,013::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71432::DEBUG::2014-09-25 17:55:53,014::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:53,027::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:53,028::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71433::DEBUG::2014-09-25 17:55:53,028::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71433::DEBUG::2014-09-25 17:55:53,030::task::595::Storage.TaskManager.Task::(_updateState) Task=`b9a7a42c-cd57-445c-bf48-0c87c489990a`::moving from state init -> state preparing >Thread-71433::INFO::2014-09-25 17:55:53,030::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71433::DEBUG::2014-09-25 17:55:53,031::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`354f2d4a-d91a-418a-af9a-534e224d66ce`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71433::DEBUG::2014-09-25 17:55:53,031::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71433::DEBUG::2014-09-25 17:55:53,031::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71433::DEBUG::2014-09-25 17:55:53,032::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`354f2d4a-d91a-418a-af9a-534e224d66ce`::Granted request >Thread-71433::DEBUG::2014-09-25 17:55:53,032::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b9a7a42c-cd57-445c-bf48-0c87c489990a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71433::DEBUG::2014-09-25 17:55:53,033::task::993::Storage.TaskManager.Task::(_decref) Task=`b9a7a42c-cd57-445c-bf48-0c87c489990a`::ref 1 aborting False >Thread-71433::INFO::2014-09-25 17:55:53,037::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71433::DEBUG::2014-09-25 17:55:53,037::task::1191::Storage.TaskManager.Task::(prepare) Task=`b9a7a42c-cd57-445c-bf48-0c87c489990a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71433::DEBUG::2014-09-25 17:55:53,037::task::595::Storage.TaskManager.Task::(_updateState) Task=`b9a7a42c-cd57-445c-bf48-0c87c489990a`::moving from state preparing -> state finished >Thread-71433::DEBUG::2014-09-25 17:55:53,038::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71433::DEBUG::2014-09-25 17:55:53,038::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71433::DEBUG::2014-09-25 17:55:53,038::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71433::DEBUG::2014-09-25 17:55:53,038::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71433::DEBUG::2014-09-25 17:55:53,039::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71433::DEBUG::2014-09-25 17:55:53,039::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71433::DEBUG::2014-09-25 17:55:53,039::task::993::Storage.TaskManager.Task::(_decref) Task=`b9a7a42c-cd57-445c-bf48-0c87c489990a`::ref 0 aborting False >Thread-71433::DEBUG::2014-09-25 17:55:53,040::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71433::DEBUG::2014-09-25 17:55:53,040::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:53,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:53,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132604 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:54,601::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:54,602::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71434::DEBUG::2014-09-25 17:55:54,604::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:55:55,316::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:55,317::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:55,317::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:55:55,318::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:55:55,319::__init__::232::IOProcess::(_processLogs) (109905) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:55,327::__init__::232::IOProcess::(_processLogs) (109905) Queuing response >Thread-81::DEBUG::2014-09-25 17:55:55,328::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000423188 s, 877 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:55,329::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:55,329::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:55,330::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:55,330::__init__::232::IOProcess::(_processLogs) (109906) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:55,331::__init__::232::IOProcess::(_processLogs) (109906) Queuing response >Dummy-148::DEBUG::2014-09-25 17:55:55,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:56,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130433 s, 78.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:57,387::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:57,387::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:57,387::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:57,388::__init__::232::IOProcess::(_processLogs) (109907) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:57,388::__init__::232::IOProcess::(_processLogs) (109907) Queuing response >Thread-80::DEBUG::2014-09-25 17:55:57,389::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:55:57,397::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000441883 s, 855 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:55:57,398::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:55:57,399::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:55:57,399::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:55:57,399::__init__::232::IOProcess::(_processLogs) (109908) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:55:57,400::__init__::232::IOProcess::(_processLogs) (109908) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:55:57,619::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:55:57,620::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71435::DEBUG::2014-09-25 17:55:57,622::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:55:58,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:55:58,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134668 s, 76.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:56:00,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:00,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0139986 s, 73.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:00,636::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:00,637::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71436::DEBUG::2014-09-25 17:56:00,640::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5e7636c-6810-4aa1-8aba-20e09702a4df`::moving from state init -> state preparing >Thread-71436::INFO::2014-09-25 17:56:00,640::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71436::INFO::2014-09-25 17:56:00,641::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000484509', 'lastCheck': '9.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000441883', 'lastCheck': '3.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000423188', 'lastCheck': '5.3', 'valid': True}} >Thread-71436::DEBUG::2014-09-25 17:56:00,641::task::1191::Storage.TaskManager.Task::(prepare) Task=`a5e7636c-6810-4aa1-8aba-20e09702a4df`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000484509', 'lastCheck': '9.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000441883', 'lastCheck': '3.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000423188', 'lastCheck': '5.3', 'valid': True}} >Thread-71436::DEBUG::2014-09-25 17:56:00,641::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5e7636c-6810-4aa1-8aba-20e09702a4df`::moving from state preparing -> state finished >Thread-71436::DEBUG::2014-09-25 17:56:00,642::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71436::DEBUG::2014-09-25 17:56:00,642::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71436::DEBUG::2014-09-25 17:56:00,642::task::993::Storage.TaskManager.Task::(_decref) Task=`a5e7636c-6810-4aa1-8aba-20e09702a4df`::ref 0 aborting False >Thread-71436::DEBUG::2014-09-25 17:56:00,644::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:00,661::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:00,662::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71437::DEBUG::2014-09-25 17:56:00,664::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:56:01,104::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:01,104::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:01,105::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:01,105::__init__::232::IOProcess::(_processLogs) (109909) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:01,106::__init__::232::IOProcess::(_processLogs) (109909) Queuing response >Thread-79::DEBUG::2014-09-25 17:56:01,107::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:56:01,127::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00174439 s, 322 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:01,128::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:01,129::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:01,129::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:01,129::__init__::232::IOProcess::(_processLogs) (109910) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:01,132::__init__::232::IOProcess::(_processLogs) (109910) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:01,134::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:01,134::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:01,134::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:01,135::__init__::232::IOProcess::(_processLogs) (109911) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:01,135::__init__::232::IOProcess::(_processLogs) (109911) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:01,136::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:01,136::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:01,137::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:01,137::__init__::232::IOProcess::(_processLogs) (109912) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:01,137::__init__::232::IOProcess::(_processLogs) (109912) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:01,138::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:01,138::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:01,139::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:01,140::__init__::232::IOProcess::(_processLogs) (109913) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:01,140::__init__::232::IOProcess::(_processLogs) (109913) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:02,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:02,135::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131785 s, 77.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:03,072::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71438::DEBUG::2014-09-25 17:56:03,073::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:56:03,073::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71438::DEBUG::2014-09-25 17:56:03,075::task::595::Storage.TaskManager.Task::(_updateState) Task=`dfe58dbb-ede5-449a-a7b7-eb800e18a08f`::moving from state init -> state preparing >Thread-71438::INFO::2014-09-25 17:56:03,076::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71438::INFO::2014-09-25 17:56:03,076::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71438::DEBUG::2014-09-25 17:56:03,077::task::1191::Storage.TaskManager.Task::(prepare) Task=`dfe58dbb-ede5-449a-a7b7-eb800e18a08f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71438::DEBUG::2014-09-25 17:56:03,077::task::595::Storage.TaskManager.Task::(_updateState) Task=`dfe58dbb-ede5-449a-a7b7-eb800e18a08f`::moving from state preparing -> state finished >Thread-71438::DEBUG::2014-09-25 17:56:03,077::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71438::DEBUG::2014-09-25 17:56:03,077::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71438::DEBUG::2014-09-25 17:56:03,078::task::993::Storage.TaskManager.Task::(_decref) Task=`dfe58dbb-ede5-449a-a7b7-eb800e18a08f`::ref 0 aborting False >Thread-71438::DEBUG::2014-09-25 17:56:03,078::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71438::DEBUG::2014-09-25 17:56:03,078::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:03,091::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:03,092::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71439::DEBUG::2014-09-25 17:56:03,092::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71439::DEBUG::2014-09-25 17:56:03,094::task::595::Storage.TaskManager.Task::(_updateState) Task=`c5057da7-c065-4a76-b2aa-79635dd23ab6`::moving from state init -> state preparing >Thread-71439::INFO::2014-09-25 17:56:03,094::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71439::DEBUG::2014-09-25 17:56:03,095::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`83f95a4c-6808-4e33-bfc7-04f634c753cf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71439::DEBUG::2014-09-25 17:56:03,095::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71439::DEBUG::2014-09-25 17:56:03,096::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71439::DEBUG::2014-09-25 17:56:03,096::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`83f95a4c-6808-4e33-bfc7-04f634c753cf`::Granted request >Thread-71439::DEBUG::2014-09-25 17:56:03,096::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c5057da7-c065-4a76-b2aa-79635dd23ab6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71439::DEBUG::2014-09-25 17:56:03,097::task::993::Storage.TaskManager.Task::(_decref) Task=`c5057da7-c065-4a76-b2aa-79635dd23ab6`::ref 1 aborting False >Thread-71439::INFO::2014-09-25 17:56:03,101::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71439::DEBUG::2014-09-25 17:56:03,101::task::1191::Storage.TaskManager.Task::(prepare) Task=`c5057da7-c065-4a76-b2aa-79635dd23ab6`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71439::DEBUG::2014-09-25 17:56:03,101::task::595::Storage.TaskManager.Task::(_updateState) Task=`c5057da7-c065-4a76-b2aa-79635dd23ab6`::moving from state preparing -> state finished >Thread-71439::DEBUG::2014-09-25 17:56:03,102::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71439::DEBUG::2014-09-25 17:56:03,102::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71439::DEBUG::2014-09-25 17:56:03,102::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71439::DEBUG::2014-09-25 17:56:03,103::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71439::DEBUG::2014-09-25 17:56:03,103::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71439::DEBUG::2014-09-25 17:56:03,103::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71439::DEBUG::2014-09-25 17:56:03,103::task::993::Storage.TaskManager.Task::(_decref) Task=`c5057da7-c065-4a76-b2aa-79635dd23ab6`::ref 0 aborting False >Thread-71439::DEBUG::2014-09-25 17:56:03,104::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71439::DEBUG::2014-09-25 17:56:03,104::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:03,716::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:03,718::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71440::DEBUG::2014-09-25 17:56:03,719::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:04,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:04,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135969 s, 75.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:05,333::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:05,333::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:05,333::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:05,334::__init__::232::IOProcess::(_processLogs) (109914) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:05,334::__init__::232::IOProcess::(_processLogs) (109914) Queuing response >Thread-81::DEBUG::2014-09-25 17:56:05,335::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:56:05,344::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000396665 s, 935 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:05,346::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:05,346::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:05,346::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:05,346::__init__::232::IOProcess::(_processLogs) (109915) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:05,347::__init__::232::IOProcess::(_processLogs) (109915) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:06,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:06,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130783 s, 78.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:06,734::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:06,735::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71441::DEBUG::2014-09-25 17:56:06,737::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:56:07,402::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:07,402::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:07,403::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:07,403::__init__::232::IOProcess::(_processLogs) (109916) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:07,403::__init__::232::IOProcess::(_processLogs) (109916) Queuing response >Thread-80::DEBUG::2014-09-25 17:56:07,404::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:56:07,414::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000517553 s, 730 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:07,415::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:07,415::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:07,416::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:07,416::__init__::232::IOProcess::(_processLogs) (109917) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:07,416::__init__::232::IOProcess::(_processLogs) (109917) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:08,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:08,261::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130256 s, 78.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:09,751::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:09,752::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71442::DEBUG::2014-09-25 17:56:09,754::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:10,281::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:10,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150505 s, 68.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:11,140::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:11,140::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:11,141::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:56:11,142::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:56:11,142::__init__::232::IOProcess::(_processLogs) (109918) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:11,151::__init__::232::IOProcess::(_processLogs) (109918) Queuing response >Thread-79::DEBUG::2014-09-25 17:56:11,152::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000410938 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:11,153::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:11,153::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:11,154::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:11,154::__init__::232::IOProcess::(_processLogs) (109919) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:11,154::__init__::232::IOProcess::(_processLogs) (109919) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:11,156::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:11,156::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:11,156::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:11,156::__init__::232::IOProcess::(_processLogs) (109920) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:11,157::__init__::232::IOProcess::(_processLogs) (109920) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:11,158::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:11,158::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:11,159::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:11,159::__init__::232::IOProcess::(_processLogs) (109921) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:11,159::__init__::232::IOProcess::(_processLogs) (109921) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:11,160::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:11,160::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:11,161::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:11,162::__init__::232::IOProcess::(_processLogs) (109922) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:11,162::__init__::232::IOProcess::(_processLogs) (109922) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:12,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:12,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133815 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:12,765::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:12,767::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71443::DEBUG::2014-09-25 17:56:12,769::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:13,132::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71444::DEBUG::2014-09-25 17:56:13,133::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:56:13,133::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71444::DEBUG::2014-09-25 17:56:13,136::task::595::Storage.TaskManager.Task::(_updateState) Task=`69bd453f-fe0c-49c9-b0b9-6febd281a87d`::moving from state init -> state preparing >Thread-71444::INFO::2014-09-25 17:56:13,136::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71444::INFO::2014-09-25 17:56:13,137::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71444::DEBUG::2014-09-25 17:56:13,137::task::1191::Storage.TaskManager.Task::(prepare) Task=`69bd453f-fe0c-49c9-b0b9-6febd281a87d`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71444::DEBUG::2014-09-25 17:56:13,137::task::595::Storage.TaskManager.Task::(_updateState) Task=`69bd453f-fe0c-49c9-b0b9-6febd281a87d`::moving from state preparing -> state finished >Thread-71444::DEBUG::2014-09-25 17:56:13,137::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71444::DEBUG::2014-09-25 17:56:13,138::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71444::DEBUG::2014-09-25 17:56:13,138::task::993::Storage.TaskManager.Task::(_decref) Task=`69bd453f-fe0c-49c9-b0b9-6febd281a87d`::ref 0 aborting False >Thread-71444::DEBUG::2014-09-25 17:56:13,138::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71444::DEBUG::2014-09-25 17:56:13,139::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:13,149::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:13,151::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71445::DEBUG::2014-09-25 17:56:13,151::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71445::DEBUG::2014-09-25 17:56:13,152::task::595::Storage.TaskManager.Task::(_updateState) Task=`1e6c93de-d0c2-40df-94e0-bde37e7671fe`::moving from state init -> state preparing >Thread-71445::INFO::2014-09-25 17:56:13,153::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71445::DEBUG::2014-09-25 17:56:13,153::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2723d6bb-ff94-41f3-81cf-25210c0cd742`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71445::DEBUG::2014-09-25 17:56:13,154::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71445::DEBUG::2014-09-25 17:56:13,154::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71445::DEBUG::2014-09-25 17:56:13,154::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2723d6bb-ff94-41f3-81cf-25210c0cd742`::Granted request >Thread-71445::DEBUG::2014-09-25 17:56:13,155::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1e6c93de-d0c2-40df-94e0-bde37e7671fe`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71445::DEBUG::2014-09-25 17:56:13,155::task::993::Storage.TaskManager.Task::(_decref) Task=`1e6c93de-d0c2-40df-94e0-bde37e7671fe`::ref 1 aborting False >Thread-71445::INFO::2014-09-25 17:56:13,160::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71445::DEBUG::2014-09-25 17:56:13,160::task::1191::Storage.TaskManager.Task::(prepare) Task=`1e6c93de-d0c2-40df-94e0-bde37e7671fe`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71445::DEBUG::2014-09-25 17:56:13,160::task::595::Storage.TaskManager.Task::(_updateState) Task=`1e6c93de-d0c2-40df-94e0-bde37e7671fe`::moving from state preparing -> state finished >Thread-71445::DEBUG::2014-09-25 17:56:13,160::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71445::DEBUG::2014-09-25 17:56:13,161::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71445::DEBUG::2014-09-25 17:56:13,161::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71445::DEBUG::2014-09-25 17:56:13,161::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71445::DEBUG::2014-09-25 17:56:13,162::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71445::DEBUG::2014-09-25 17:56:13,162::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71445::DEBUG::2014-09-25 17:56:13,162::task::993::Storage.TaskManager.Task::(_decref) Task=`1e6c93de-d0c2-40df-94e0-bde37e7671fe`::ref 0 aborting False >Thread-71445::DEBUG::2014-09-25 17:56:13,163::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71445::DEBUG::2014-09-25 17:56:13,163::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:14,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:14,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132949 s, 77.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:15,349::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:15,350::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:15,350::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:15,350::__init__::232::IOProcess::(_processLogs) (109923) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:15,350::__init__::232::IOProcess::(_processLogs) (109923) Queuing response >Thread-81::DEBUG::2014-09-25 17:56:15,352::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:56:15,361::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000478583 s, 775 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:15,362::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:15,362::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:15,362::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:15,363::__init__::232::IOProcess::(_processLogs) (109924) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:15,363::__init__::232::IOProcess::(_processLogs) (109924) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:15,781::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:15,782::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71446::DEBUG::2014-09-25 17:56:15,785::task::595::Storage.TaskManager.Task::(_updateState) Task=`d02f235a-a4eb-42f2-979b-bdf1a21c3616`::moving from state init -> state preparing >Thread-71446::INFO::2014-09-25 17:56:15,785::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71446::INFO::2014-09-25 17:56:15,786::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000410938', 'lastCheck': '4.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000517553', 'lastCheck': '8.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000478583', 'lastCheck': '0.4', 'valid': True}} >Thread-71446::DEBUG::2014-09-25 17:56:15,786::task::1191::Storage.TaskManager.Task::(prepare) Task=`d02f235a-a4eb-42f2-979b-bdf1a21c3616`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000410938', 'lastCheck': '4.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000517553', 'lastCheck': '8.4', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000478583', 'lastCheck': '0.4', 'valid': True}} >Thread-71446::DEBUG::2014-09-25 17:56:15,786::task::595::Storage.TaskManager.Task::(_updateState) Task=`d02f235a-a4eb-42f2-979b-bdf1a21c3616`::moving from state preparing -> state finished >Thread-71446::DEBUG::2014-09-25 17:56:15,787::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71446::DEBUG::2014-09-25 17:56:15,787::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71446::DEBUG::2014-09-25 17:56:15,787::task::993::Storage.TaskManager.Task::(_decref) Task=`d02f235a-a4eb-42f2-979b-bdf1a21c3616`::ref 0 aborting False >Thread-71446::DEBUG::2014-09-25 17:56:15,790::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:15,805::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:15,807::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71447::DEBUG::2014-09-25 17:56:15,809::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:16,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:16,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.022225 s, 46.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:17,419::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:17,419::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:17,419::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:17,419::__init__::232::IOProcess::(_processLogs) (109925) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:17,419::__init__::232::IOProcess::(_processLogs) (109925) Queuing response >Thread-80::DEBUG::2014-09-25 17:56:17,420::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:56:17,427::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000480091 s, 787 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:17,428::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:17,428::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:17,428::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:17,428::__init__::232::IOProcess::(_processLogs) (109926) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:17,429::__init__::232::IOProcess::(_processLogs) (109926) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:18,618::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:18,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013557 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:18,857::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:18,859::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71448::DEBUG::2014-09-25 17:56:18,861::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:20,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:20,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171462 s, 59.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:21,162::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:21,162::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:21,163::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:21,163::__init__::232::IOProcess::(_processLogs) (109927) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:21,174::__init__::232::IOProcess::(_processLogs) (109927) Queuing response >Thread-79::DEBUG::2014-09-25 17:56:21,175::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:56:21,213::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00765369 s, 73.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:21,215::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:21,215::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:21,215::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:21,215::__init__::232::IOProcess::(_processLogs) (109928) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:21,223::__init__::232::IOProcess::(_processLogs) (109928) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:21,224::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:21,225::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:21,225::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:21,226::__init__::232::IOProcess::(_processLogs) (109929) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:21,226::__init__::232::IOProcess::(_processLogs) (109929) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:21,226::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:21,227::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:21,227::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:21,228::__init__::232::IOProcess::(_processLogs) (109930) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:21,228::__init__::232::IOProcess::(_processLogs) (109930) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:21,228::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:21,229::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:21,230::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:21,230::__init__::232::IOProcess::(_processLogs) (109931) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:21,230::__init__::232::IOProcess::(_processLogs) (109931) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:21,874::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:21,875::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71449::DEBUG::2014-09-25 17:56:21,877::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:22,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:22,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136128 s, 75.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:23,191::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:23,193::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71450::DEBUG::2014-09-25 17:56:23,193::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71450::DEBUG::2014-09-25 17:56:23,195::task::595::Storage.TaskManager.Task::(_updateState) Task=`666ab7a9-fdc1-4074-971f-8ec1281b0878`::moving from state init -> state preparing >Thread-71450::INFO::2014-09-25 17:56:23,195::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71450::INFO::2014-09-25 17:56:23,196::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71450::DEBUG::2014-09-25 17:56:23,196::task::1191::Storage.TaskManager.Task::(prepare) Task=`666ab7a9-fdc1-4074-971f-8ec1281b0878`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71450::DEBUG::2014-09-25 17:56:23,196::task::595::Storage.TaskManager.Task::(_updateState) Task=`666ab7a9-fdc1-4074-971f-8ec1281b0878`::moving from state preparing -> state finished >Thread-71450::DEBUG::2014-09-25 17:56:23,196::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71450::DEBUG::2014-09-25 17:56:23,196::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71450::DEBUG::2014-09-25 17:56:23,197::task::993::Storage.TaskManager.Task::(_decref) Task=`666ab7a9-fdc1-4074-971f-8ec1281b0878`::ref 0 aborting False >Thread-71450::DEBUG::2014-09-25 17:56:23,197::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71450::DEBUG::2014-09-25 17:56:23,197::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:23,212::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:23,213::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71451::DEBUG::2014-09-25 17:56:23,214::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71451::DEBUG::2014-09-25 17:56:23,215::task::595::Storage.TaskManager.Task::(_updateState) Task=`b90931dc-5954-4b5f-a3f0-036016b22fba`::moving from state init -> state preparing >Thread-71451::INFO::2014-09-25 17:56:23,215::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71451::DEBUG::2014-09-25 17:56:23,216::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`14042508-c476-4326-9d72-2881e84361d3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71451::DEBUG::2014-09-25 17:56:23,216::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71451::DEBUG::2014-09-25 17:56:23,217::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71451::DEBUG::2014-09-25 17:56:23,217::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`14042508-c476-4326-9d72-2881e84361d3`::Granted request >Thread-71451::DEBUG::2014-09-25 17:56:23,219::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b90931dc-5954-4b5f-a3f0-036016b22fba`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71451::DEBUG::2014-09-25 17:56:23,219::task::993::Storage.TaskManager.Task::(_decref) Task=`b90931dc-5954-4b5f-a3f0-036016b22fba`::ref 1 aborting False >Thread-71451::INFO::2014-09-25 17:56:23,223::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71451::DEBUG::2014-09-25 17:56:23,224::task::1191::Storage.TaskManager.Task::(prepare) Task=`b90931dc-5954-4b5f-a3f0-036016b22fba`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71451::DEBUG::2014-09-25 17:56:23,224::task::595::Storage.TaskManager.Task::(_updateState) Task=`b90931dc-5954-4b5f-a3f0-036016b22fba`::moving from state preparing -> state finished >Thread-71451::DEBUG::2014-09-25 17:56:23,224::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71451::DEBUG::2014-09-25 17:56:23,224::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71451::DEBUG::2014-09-25 17:56:23,225::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71451::DEBUG::2014-09-25 17:56:23,225::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71451::DEBUG::2014-09-25 17:56:23,225::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71451::DEBUG::2014-09-25 17:56:23,225::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71451::DEBUG::2014-09-25 17:56:23,226::task::993::Storage.TaskManager.Task::(_decref) Task=`b90931dc-5954-4b5f-a3f0-036016b22fba`::ref 0 aborting False >Thread-71451::DEBUG::2014-09-25 17:56:23,226::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71451::DEBUG::2014-09-25 17:56:23,227::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:24,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:24,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0126812 s, 80.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:24,892::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:24,893::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71452::DEBUG::2014-09-25 17:56:24,895::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:56:25,365::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:25,366::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:25,366::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:56:25,367::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:56:25,368::__init__::232::IOProcess::(_processLogs) (109932) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:25,376::__init__::232::IOProcess::(_processLogs) (109932) Queuing response >Thread-81::DEBUG::2014-09-25 17:56:25,377::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000384525 s, 965 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:25,378::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:25,378::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:25,379::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:25,379::__init__::232::IOProcess::(_processLogs) (109933) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:25,380::__init__::232::IOProcess::(_processLogs) (109933) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:26,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:26,815::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133896 s, 76.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:27,430::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:27,431::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:27,431::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:27,431::__init__::232::IOProcess::(_processLogs) (109934) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:27,432::__init__::232::IOProcess::(_processLogs) (109934) Queuing response >Thread-80::DEBUG::2014-09-25 17:56:27,433::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:56:27,442::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000517544 s, 730 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:27,443::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:27,443::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:27,443::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:27,444::__init__::232::IOProcess::(_processLogs) (109935) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:27,444::__init__::232::IOProcess::(_processLogs) (109935) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:27,910::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:27,911::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71453::DEBUG::2014-09-25 17:56:27,913::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:28,835::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:28,857::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013422 s, 76.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:56:30,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:30,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137157 s, 74.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:30,925::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:30,927::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71454::DEBUG::2014-09-25 17:56:30,930::task::595::Storage.TaskManager.Task::(_updateState) Task=`21547cb4-b963-4b80-afcb-529a0adf0e08`::moving from state init -> state preparing >Thread-71454::INFO::2014-09-25 17:56:30,930::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71454::INFO::2014-09-25 17:56:30,930::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00765369', 'lastCheck': '9.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000517544', 'lastCheck': '3.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000384525', 'lastCheck': '5.5', 'valid': True}} >Thread-71454::DEBUG::2014-09-25 17:56:30,931::task::1191::Storage.TaskManager.Task::(prepare) Task=`21547cb4-b963-4b80-afcb-529a0adf0e08`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00765369', 'lastCheck': '9.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000517544', 'lastCheck': '3.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000384525', 'lastCheck': '5.5', 'valid': True}} >Thread-71454::DEBUG::2014-09-25 17:56:30,931::task::595::Storage.TaskManager.Task::(_updateState) Task=`21547cb4-b963-4b80-afcb-529a0adf0e08`::moving from state preparing -> state finished >Thread-71454::DEBUG::2014-09-25 17:56:30,931::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71454::DEBUG::2014-09-25 17:56:30,931::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71454::DEBUG::2014-09-25 17:56:30,932::task::993::Storage.TaskManager.Task::(_decref) Task=`21547cb4-b963-4b80-afcb-529a0adf0e08`::ref 0 aborting False >Thread-71454::DEBUG::2014-09-25 17:56:30,934::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:30,948::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:30,949::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71455::DEBUG::2014-09-25 17:56:30,951::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:56:31,230::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:31,231::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:31,231::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:31,231::__init__::232::IOProcess::(_processLogs) (109936) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:31,231::__init__::232::IOProcess::(_processLogs) (109936) Queuing response >Thread-79::DEBUG::2014-09-25 17:56:31,233::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:56:31,241::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000460026 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:31,243::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:31,243::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:31,244::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:31,245::__init__::232::IOProcess::(_processLogs) (109937) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:31,245::__init__::232::IOProcess::(_processLogs) (109937) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:31,246::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:31,246::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:31,246::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:31,247::__init__::232::IOProcess::(_processLogs) (109938) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:31,247::__init__::232::IOProcess::(_processLogs) (109938) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:31,248::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:31,248::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:31,248::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:31,249::__init__::232::IOProcess::(_processLogs) (109939) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:31,249::__init__::232::IOProcess::(_processLogs) (109939) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:31,250::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:31,250::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:31,250::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:31,251::__init__::232::IOProcess::(_processLogs) (109940) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:31,252::__init__::232::IOProcess::(_processLogs) (109940) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:32,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:32,942::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144921 s, 70.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:33,261::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71456::DEBUG::2014-09-25 17:56:33,262::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:56:33,262::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71456::DEBUG::2014-09-25 17:56:33,265::task::595::Storage.TaskManager.Task::(_updateState) Task=`7cd35c73-f73b-463a-87b4-71426b0b715a`::moving from state init -> state preparing >Thread-71456::INFO::2014-09-25 17:56:33,265::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71456::INFO::2014-09-25 17:56:33,265::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71456::DEBUG::2014-09-25 17:56:33,266::task::1191::Storage.TaskManager.Task::(prepare) Task=`7cd35c73-f73b-463a-87b4-71426b0b715a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71456::DEBUG::2014-09-25 17:56:33,266::task::595::Storage.TaskManager.Task::(_updateState) Task=`7cd35c73-f73b-463a-87b4-71426b0b715a`::moving from state preparing -> state finished >Thread-71456::DEBUG::2014-09-25 17:56:33,266::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71456::DEBUG::2014-09-25 17:56:33,266::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71456::DEBUG::2014-09-25 17:56:33,267::task::993::Storage.TaskManager.Task::(_decref) Task=`7cd35c73-f73b-463a-87b4-71426b0b715a`::ref 0 aborting False >Thread-71456::DEBUG::2014-09-25 17:56:33,267::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71456::DEBUG::2014-09-25 17:56:33,267::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:33,279::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:33,280::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71457::DEBUG::2014-09-25 17:56:33,281::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71457::DEBUG::2014-09-25 17:56:33,282::task::595::Storage.TaskManager.Task::(_updateState) Task=`b2a2cc29-aea8-4812-9a6d-f0742e36dc11`::moving from state init -> state preparing >Thread-71457::INFO::2014-09-25 17:56:33,283::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71457::DEBUG::2014-09-25 17:56:33,283::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ba7ca3ef-6ba4-417a-91bf-5005b1f981d0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71457::DEBUG::2014-09-25 17:56:33,283::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71457::DEBUG::2014-09-25 17:56:33,284::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71457::DEBUG::2014-09-25 17:56:33,284::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ba7ca3ef-6ba4-417a-91bf-5005b1f981d0`::Granted request >Thread-71457::DEBUG::2014-09-25 17:56:33,285::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b2a2cc29-aea8-4812-9a6d-f0742e36dc11`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71457::DEBUG::2014-09-25 17:56:33,285::task::993::Storage.TaskManager.Task::(_decref) Task=`b2a2cc29-aea8-4812-9a6d-f0742e36dc11`::ref 1 aborting False >Thread-71457::INFO::2014-09-25 17:56:33,289::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71457::DEBUG::2014-09-25 17:56:33,290::task::1191::Storage.TaskManager.Task::(prepare) Task=`b2a2cc29-aea8-4812-9a6d-f0742e36dc11`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71457::DEBUG::2014-09-25 17:56:33,290::task::595::Storage.TaskManager.Task::(_updateState) Task=`b2a2cc29-aea8-4812-9a6d-f0742e36dc11`::moving from state preparing -> state finished >Thread-71457::DEBUG::2014-09-25 17:56:33,290::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71457::DEBUG::2014-09-25 17:56:33,290::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71457::DEBUG::2014-09-25 17:56:33,291::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71457::DEBUG::2014-09-25 17:56:33,291::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71457::DEBUG::2014-09-25 17:56:33,291::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71457::DEBUG::2014-09-25 17:56:33,291::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71457::DEBUG::2014-09-25 17:56:33,292::task::993::Storage.TaskManager.Task::(_decref) Task=`b2a2cc29-aea8-4812-9a6d-f0742e36dc11`::ref 0 aborting False >Thread-71457::DEBUG::2014-09-25 17:56:33,292::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71457::DEBUG::2014-09-25 17:56:33,293::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:33,995::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:33,996::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71458::DEBUG::2014-09-25 17:56:33,998::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:34,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:34,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134295 s, 76.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:35,381::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:35,382::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:35,382::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:35,382::__init__::232::IOProcess::(_processLogs) (109941) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:35,383::__init__::232::IOProcess::(_processLogs) (109941) Queuing response >Thread-81::DEBUG::2014-09-25 17:56:35,384::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:56:35,393::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000505031 s, 735 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:35,394::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:35,395::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:35,395::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:35,395::__init__::232::IOProcess::(_processLogs) (109942) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:35,396::__init__::232::IOProcess::(_processLogs) (109942) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:37,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:37,015::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:37,018::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71459::DEBUG::2014-09-25 17:56:37,021::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:37,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132795 s, 77.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:37,446::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:37,447::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:37,447::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:37,447::__init__::232::IOProcess::(_processLogs) (109943) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:37,448::__init__::232::IOProcess::(_processLogs) (109943) Queuing response >Thread-80::DEBUG::2014-09-25 17:56:37,449::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:56:37,458::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000409527 s, 923 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:37,459::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:37,459::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:37,459::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:37,460::__init__::232::IOProcess::(_processLogs) (109944) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:37,460::__init__::232::IOProcess::(_processLogs) (109944) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:39,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:39,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135145 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:40,035::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:40,037::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71460::DEBUG::2014-09-25 17:56:40,039::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:41,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:41,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129899 s, 78.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:41,252::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:41,253::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:41,253::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:41,253::__init__::232::IOProcess::(_processLogs) (109945) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:41,253::__init__::232::IOProcess::(_processLogs) (109945) Queuing response >Thread-79::DEBUG::2014-09-25 17:56:41,254::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:56:41,263::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000464356 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:41,264::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:41,264::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:41,265::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:41,265::__init__::232::IOProcess::(_processLogs) (109946) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:41,265::__init__::232::IOProcess::(_processLogs) (109946) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:41,267::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:41,267::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:41,268::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:41,268::__init__::232::IOProcess::(_processLogs) (109947) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:41,268::__init__::232::IOProcess::(_processLogs) (109947) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:41,269::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:41,269::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:41,270::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:41,270::__init__::232::IOProcess::(_processLogs) (109948) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:41,270::__init__::232::IOProcess::(_processLogs) (109948) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:41,271::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:41,271::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:41,272::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:41,272::__init__::232::IOProcess::(_processLogs) (109949) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:41,273::__init__::232::IOProcess::(_processLogs) (109949) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:43,049::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:43,051::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71461::DEBUG::2014-09-25 17:56:43,053::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:43,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:43,155::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133083 s, 76.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:43,318::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:43,320::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71462::DEBUG::2014-09-25 17:56:43,320::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71462::DEBUG::2014-09-25 17:56:43,322::task::595::Storage.TaskManager.Task::(_updateState) Task=`b80e4225-c4a9-45d6-afa1-b67e137167ee`::moving from state init -> state preparing >Thread-71462::INFO::2014-09-25 17:56:43,323::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71462::INFO::2014-09-25 17:56:43,323::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71462::DEBUG::2014-09-25 17:56:43,324::task::1191::Storage.TaskManager.Task::(prepare) Task=`b80e4225-c4a9-45d6-afa1-b67e137167ee`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71462::DEBUG::2014-09-25 17:56:43,324::task::595::Storage.TaskManager.Task::(_updateState) Task=`b80e4225-c4a9-45d6-afa1-b67e137167ee`::moving from state preparing -> state finished >Thread-71462::DEBUG::2014-09-25 17:56:43,324::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71462::DEBUG::2014-09-25 17:56:43,324::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71462::DEBUG::2014-09-25 17:56:43,324::task::993::Storage.TaskManager.Task::(_decref) Task=`b80e4225-c4a9-45d6-afa1-b67e137167ee`::ref 0 aborting False >Thread-71462::DEBUG::2014-09-25 17:56:43,325::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71462::DEBUG::2014-09-25 17:56:43,325::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:43,336::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:43,337::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71463::DEBUG::2014-09-25 17:56:43,337::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71463::DEBUG::2014-09-25 17:56:43,339::task::595::Storage.TaskManager.Task::(_updateState) Task=`aace28c8-1d36-42fd-99a4-e937ef4f0d7f`::moving from state init -> state preparing >Thread-71463::INFO::2014-09-25 17:56:43,339::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71463::DEBUG::2014-09-25 17:56:43,340::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3c4fb88c-bd38-414f-aeec-6cdbee52e101`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71463::DEBUG::2014-09-25 17:56:43,340::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71463::DEBUG::2014-09-25 17:56:43,341::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71463::DEBUG::2014-09-25 17:56:43,341::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`3c4fb88c-bd38-414f-aeec-6cdbee52e101`::Granted request >Thread-71463::DEBUG::2014-09-25 17:56:43,342::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`aace28c8-1d36-42fd-99a4-e937ef4f0d7f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71463::DEBUG::2014-09-25 17:56:43,342::task::993::Storage.TaskManager.Task::(_decref) Task=`aace28c8-1d36-42fd-99a4-e937ef4f0d7f`::ref 1 aborting False >Thread-71463::INFO::2014-09-25 17:56:43,346::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71463::DEBUG::2014-09-25 17:56:43,347::task::1191::Storage.TaskManager.Task::(prepare) Task=`aace28c8-1d36-42fd-99a4-e937ef4f0d7f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71463::DEBUG::2014-09-25 17:56:43,347::task::595::Storage.TaskManager.Task::(_updateState) Task=`aace28c8-1d36-42fd-99a4-e937ef4f0d7f`::moving from state preparing -> state finished >Thread-71463::DEBUG::2014-09-25 17:56:43,347::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71463::DEBUG::2014-09-25 17:56:43,347::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71463::DEBUG::2014-09-25 17:56:43,348::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71463::DEBUG::2014-09-25 17:56:43,348::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71463::DEBUG::2014-09-25 17:56:43,348::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71463::DEBUG::2014-09-25 17:56:43,349::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71463::DEBUG::2014-09-25 17:56:43,349::task::993::Storage.TaskManager.Task::(_decref) Task=`aace28c8-1d36-42fd-99a4-e937ef4f0d7f`::ref 0 aborting False >Thread-71463::DEBUG::2014-09-25 17:56:43,349::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71463::DEBUG::2014-09-25 17:56:43,350::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:45,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:45,197::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135116 s, 75.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:45,398::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:45,398::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:45,399::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:45,399::__init__::232::IOProcess::(_processLogs) (109950) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:45,399::__init__::232::IOProcess::(_processLogs) (109950) Queuing response >Thread-81::DEBUG::2014-09-25 17:56:45,400::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:56:45,408::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000401674 s, 924 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:45,410::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:45,410::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:45,410::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:45,411::__init__::232::IOProcess::(_processLogs) (109951) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:45,411::__init__::232::IOProcess::(_processLogs) (109951) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:46,066::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:46,072::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71464::DEBUG::2014-09-25 17:56:46,093::task::595::Storage.TaskManager.Task::(_updateState) Task=`5f87ea87-cb08-40dc-8b9f-233259fd77e6`::moving from state init -> state preparing >Thread-71464::INFO::2014-09-25 17:56:46,096::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71464::INFO::2014-09-25 17:56:46,102::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000464356', 'lastCheck': '4.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409527', 'lastCheck': '8.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000401674', 'lastCheck': '0.7', 'valid': True}} >Thread-71464::DEBUG::2014-09-25 17:56:46,105::task::1191::Storage.TaskManager.Task::(prepare) Task=`5f87ea87-cb08-40dc-8b9f-233259fd77e6`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000464356', 'lastCheck': '4.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000409527', 'lastCheck': '8.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000401674', 'lastCheck': '0.7', 'valid': True}} >Thread-71464::DEBUG::2014-09-25 17:56:46,108::task::595::Storage.TaskManager.Task::(_updateState) Task=`5f87ea87-cb08-40dc-8b9f-233259fd77e6`::moving from state preparing -> state finished >Thread-71464::DEBUG::2014-09-25 17:56:46,110::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71464::DEBUG::2014-09-25 17:56:46,112::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71464::DEBUG::2014-09-25 17:56:46,115::task::993::Storage.TaskManager.Task::(_decref) Task=`5f87ea87-cb08-40dc-8b9f-233259fd77e6`::ref 0 aborting False >Thread-71464::DEBUG::2014-09-25 17:56:46,139::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:46,167::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:46,173::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71465::DEBUG::2014-09-25 17:56:46,186::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:47,218::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:47,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0120908 s, 84.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:47,462::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:47,463::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:47,463::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:47,463::__init__::232::IOProcess::(_processLogs) (109952) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:47,464::__init__::232::IOProcess::(_processLogs) (109952) Queuing response >Thread-80::DEBUG::2014-09-25 17:56:47,464::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:56:47,472::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000527725 s, 716 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:47,473::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:47,473::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:47,473::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:47,473::__init__::232::IOProcess::(_processLogs) (109953) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:47,474::__init__::232::IOProcess::(_processLogs) (109953) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:49,242::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:49,243::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71466::DEBUG::2014-09-25 17:56:49,245::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:49,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:49,271::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136378 s, 75.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:51,273::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:51,273::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:51,273::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:51,274::__init__::232::IOProcess::(_processLogs) (109954) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:51,274::__init__::232::IOProcess::(_processLogs) (109954) Queuing response >Thread-79::DEBUG::2014-09-25 17:56:51,275::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:56:51,283::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00039152 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:51,285::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:51,285::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:51,285::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:51,286::__init__::232::IOProcess::(_processLogs) (109955) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:51,286::__init__::232::IOProcess::(_processLogs) (109955) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:51,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:51,288::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:51,288::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:51,288::__init__::232::IOProcess::(_processLogs) (109956) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:51,289::__init__::232::IOProcess::(_processLogs) (109956) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:51,290::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:51,290::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:51,291::__init__::232::IOProcess::(_processLogs) Extracting request information... >Dummy-148::DEBUG::2014-09-25 17:56:51,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 17:56:51,292::__init__::232::IOProcess::(_processLogs) (109957) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:51,301::__init__::232::IOProcess::(_processLogs) (109957) Queuing response >Thread-29::DEBUG::2014-09-25 17:56:51,302::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:51,302::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:51,302::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:51,302::__init__::232::IOProcess::(_processLogs) (109958) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:56:51,310::__init__::232::IOProcess::(_processLogs) (109958) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:51,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150082 s, 68.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:52,258::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:52,260::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71467::DEBUG::2014-09-25 17:56:52,262::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:53,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:53,357::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130779 s, 78.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:53,375::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:53,378::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71468::DEBUG::2014-09-25 17:56:53,379::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71468::DEBUG::2014-09-25 17:56:53,381::task::595::Storage.TaskManager.Task::(_updateState) Task=`0d88569f-444b-4593-b034-76149ad94a4e`::moving from state init -> state preparing >Thread-71468::INFO::2014-09-25 17:56:53,381::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71468::INFO::2014-09-25 17:56:53,382::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71468::DEBUG::2014-09-25 17:56:53,382::task::1191::Storage.TaskManager.Task::(prepare) Task=`0d88569f-444b-4593-b034-76149ad94a4e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71468::DEBUG::2014-09-25 17:56:53,382::task::595::Storage.TaskManager.Task::(_updateState) Task=`0d88569f-444b-4593-b034-76149ad94a4e`::moving from state preparing -> state finished >Thread-71468::DEBUG::2014-09-25 17:56:53,382::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71468::DEBUG::2014-09-25 17:56:53,382::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71468::DEBUG::2014-09-25 17:56:53,383::task::993::Storage.TaskManager.Task::(_decref) Task=`0d88569f-444b-4593-b034-76149ad94a4e`::ref 0 aborting False >Thread-71468::DEBUG::2014-09-25 17:56:53,383::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71468::DEBUG::2014-09-25 17:56:53,383::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:53,399::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:53,400::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71469::DEBUG::2014-09-25 17:56:53,400::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71469::DEBUG::2014-09-25 17:56:53,402::task::595::Storage.TaskManager.Task::(_updateState) Task=`12de872a-0dcd-4b47-8455-912f1b33d769`::moving from state init -> state preparing >Thread-71469::INFO::2014-09-25 17:56:53,402::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71469::DEBUG::2014-09-25 17:56:53,403::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b8d0ffbc-b9e3-4d46-be62-090038df19a8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71469::DEBUG::2014-09-25 17:56:53,403::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71469::DEBUG::2014-09-25 17:56:53,403::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71469::DEBUG::2014-09-25 17:56:53,403::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b8d0ffbc-b9e3-4d46-be62-090038df19a8`::Granted request >Thread-71469::DEBUG::2014-09-25 17:56:53,404::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`12de872a-0dcd-4b47-8455-912f1b33d769`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71469::DEBUG::2014-09-25 17:56:53,404::task::993::Storage.TaskManager.Task::(_decref) Task=`12de872a-0dcd-4b47-8455-912f1b33d769`::ref 1 aborting False >Thread-71469::INFO::2014-09-25 17:56:53,409::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71469::DEBUG::2014-09-25 17:56:53,409::task::1191::Storage.TaskManager.Task::(prepare) Task=`12de872a-0dcd-4b47-8455-912f1b33d769`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71469::DEBUG::2014-09-25 17:56:53,409::task::595::Storage.TaskManager.Task::(_updateState) Task=`12de872a-0dcd-4b47-8455-912f1b33d769`::moving from state preparing -> state finished >Thread-71469::DEBUG::2014-09-25 17:56:53,409::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71469::DEBUG::2014-09-25 17:56:53,410::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71469::DEBUG::2014-09-25 17:56:53,410::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71469::DEBUG::2014-09-25 17:56:53,410::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71469::DEBUG::2014-09-25 17:56:53,410::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71469::DEBUG::2014-09-25 17:56:53,411::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71469::DEBUG::2014-09-25 17:56:53,411::task::993::Storage.TaskManager.Task::(_decref) Task=`12de872a-0dcd-4b47-8455-912f1b33d769`::ref 0 aborting False >Thread-71469::DEBUG::2014-09-25 17:56:53,411::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7249854464', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71469::DEBUG::2014-09-25 17:56:53,412::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:55,276::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:55,277::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71470::DEBUG::2014-09-25 17:56:55,279::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:55,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:55,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132503 s, 77.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:55,414::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:55,415::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:55,416::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:55,417::__init__::232::IOProcess::(_processLogs) (109959) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:55,418::__init__::232::IOProcess::(_processLogs) (109959) Queuing response >Thread-81::DEBUG::2014-09-25 17:56:55,420::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:56:55,430::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000418006 s, 888 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:55,432::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:55,433::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:55,434::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:55,435::__init__::232::IOProcess::(_processLogs) (109960) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:55,437::__init__::232::IOProcess::(_processLogs) (109960) Queuing response >Dummy-148::DEBUG::2014-09-25 17:56:57,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:57,469::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133314 s, 76.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:57,476::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:57,478::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:57,478::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:57,479::__init__::232::IOProcess::(_processLogs) (109961) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:57,481::__init__::232::IOProcess::(_processLogs) (109961) Queuing response >Thread-80::DEBUG::2014-09-25 17:56:57,483::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:56:57,493::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000529038 s, 715 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:56:57,496::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:56:57,497::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:56:57,498::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:56:57,499::__init__::232::IOProcess::(_processLogs) (109962) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:56:57,500::__init__::232::IOProcess::(_processLogs) (109962) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:56:58,293::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:56:58,296::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71471::DEBUG::2014-09-25 17:56:58,298::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:56:59,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:56:59,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013741 s, 74.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:01,314::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:01,314::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:01,314::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:01,315::__init__::232::IOProcess::(_processLogs) (109963) Got request for method 'statvfs' >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:01,315::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-29::DEBUG::2014-09-25 17:57:01,317::__init__::232::IOProcess::(_processLogs) (109963) Queuing response >Thread-79::DEBUG::2014-09-25 17:57:01,320::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >JsonRpcServer::DEBUG::2014-09-25 17:57:01,320::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-79::DEBUG::2014-09-25 17:57:01,331::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00033676 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:01,334::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:01,336::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:01,337::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-71472::DEBUG::2014-09-25 17:57:01,338::task::595::Storage.TaskManager.Task::(_updateState) Task=`726dd03e-cd45-410a-88e3-3777bacfa39c`::moving from state init -> state preparing >Thread-29::DEBUG::2014-09-25 17:57:01,338::__init__::232::IOProcess::(_processLogs) (109964) Got request for method 'statvfs' >Thread-71472::INFO::2014-09-25 17:57:01,339::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-29::DEBUG::2014-09-25 17:57:01,340::__init__::232::IOProcess::(_processLogs) (109964) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:01,343::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-71472::INFO::2014-09-25 17:57:01,344::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00039152', 'lastCheck': '10.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000529038', 'lastCheck': '3.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000418006', 'lastCheck': '5.9', 'valid': True}} >Thread-29::DEBUG::2014-09-25 17:57:01,345::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-71472::DEBUG::2014-09-25 17:57:01,345::task::1191::Storage.TaskManager.Task::(prepare) Task=`726dd03e-cd45-410a-88e3-3777bacfa39c`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00039152', 'lastCheck': '10.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000529038', 'lastCheck': '3.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000418006', 'lastCheck': '5.9', 'valid': True}} >Thread-29::DEBUG::2014-09-25 17:57:01,346::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-71472::DEBUG::2014-09-25 17:57:01,346::task::595::Storage.TaskManager.Task::(_updateState) Task=`726dd03e-cd45-410a-88e3-3777bacfa39c`::moving from state preparing -> state finished >Thread-29::DEBUG::2014-09-25 17:57:01,347::__init__::232::IOProcess::(_processLogs) (109965) Got request for method 'access' >Thread-71472::DEBUG::2014-09-25 17:57:01,348::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-29::DEBUG::2014-09-25 17:57:01,348::__init__::232::IOProcess::(_processLogs) (109965) Queuing response >Thread-71472::DEBUG::2014-09-25 17:57:01,349::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71472::DEBUG::2014-09-25 17:57:01,350::task::993::Storage.TaskManager.Task::(_decref) Task=`726dd03e-cd45-410a-88e3-3777bacfa39c`::ref 0 aborting False >Thread-29::DEBUG::2014-09-25 17:57:01,351::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:01,352::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:01,353::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:01,355::__init__::232::IOProcess::(_processLogs) (109966) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:01,356::__init__::232::IOProcess::(_processLogs) (109966) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:01,359::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:01,360::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-71472::DEBUG::2014-09-25 17:57:01,361::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:57:01,362::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:01,363::__init__::232::IOProcess::(_processLogs) (109967) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:01,364::__init__::232::IOProcess::(_processLogs) (109967) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:01,386::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:01,388::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71473::DEBUG::2014-09-25 17:57:01,389::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:01,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:01,578::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132548 s, 77.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:03,441::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71474::DEBUG::2014-09-25 17:57:03,442::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:57:03,442::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71474::DEBUG::2014-09-25 17:57:03,444::task::595::Storage.TaskManager.Task::(_updateState) Task=`f439fd6b-f2ec-459f-baed-277d3b64555e`::moving from state init -> state preparing >Thread-71474::INFO::2014-09-25 17:57:03,445::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71474::INFO::2014-09-25 17:57:03,445::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71474::DEBUG::2014-09-25 17:57:03,446::task::1191::Storage.TaskManager.Task::(prepare) Task=`f439fd6b-f2ec-459f-baed-277d3b64555e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71474::DEBUG::2014-09-25 17:57:03,446::task::595::Storage.TaskManager.Task::(_updateState) Task=`f439fd6b-f2ec-459f-baed-277d3b64555e`::moving from state preparing -> state finished >Thread-71474::DEBUG::2014-09-25 17:57:03,446::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71474::DEBUG::2014-09-25 17:57:03,446::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71474::DEBUG::2014-09-25 17:57:03,446::task::993::Storage.TaskManager.Task::(_decref) Task=`f439fd6b-f2ec-459f-baed-277d3b64555e`::ref 0 aborting False >Thread-71474::DEBUG::2014-09-25 17:57:03,447::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71474::DEBUG::2014-09-25 17:57:03,447::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:03,458::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:03,460::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71475::DEBUG::2014-09-25 17:57:03,460::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71475::DEBUG::2014-09-25 17:57:03,461::task::595::Storage.TaskManager.Task::(_updateState) Task=`ac43239e-c175-4353-9ea6-2d1ea59dea66`::moving from state init -> state preparing >Thread-71475::INFO::2014-09-25 17:57:03,462::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71475::DEBUG::2014-09-25 17:57:03,462::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`63dd0f11-d1d4-4d2d-b85d-6974015e2cfa`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71475::DEBUG::2014-09-25 17:57:03,463::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71475::DEBUG::2014-09-25 17:57:03,463::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71475::DEBUG::2014-09-25 17:57:03,463::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`63dd0f11-d1d4-4d2d-b85d-6974015e2cfa`::Granted request >Thread-71475::DEBUG::2014-09-25 17:57:03,465::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ac43239e-c175-4353-9ea6-2d1ea59dea66`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71475::DEBUG::2014-09-25 17:57:03,465::task::993::Storage.TaskManager.Task::(_decref) Task=`ac43239e-c175-4353-9ea6-2d1ea59dea66`::ref 1 aborting False >Thread-71475::INFO::2014-09-25 17:57:03,469::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7238320128', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71475::DEBUG::2014-09-25 17:57:03,470::task::1191::Storage.TaskManager.Task::(prepare) Task=`ac43239e-c175-4353-9ea6-2d1ea59dea66`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7238320128', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71475::DEBUG::2014-09-25 17:57:03,470::task::595::Storage.TaskManager.Task::(_updateState) Task=`ac43239e-c175-4353-9ea6-2d1ea59dea66`::moving from state preparing -> state finished >Thread-71475::DEBUG::2014-09-25 17:57:03,470::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71475::DEBUG::2014-09-25 17:57:03,470::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71475::DEBUG::2014-09-25 17:57:03,471::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71475::DEBUG::2014-09-25 17:57:03,471::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71475::DEBUG::2014-09-25 17:57:03,471::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71475::DEBUG::2014-09-25 17:57:03,472::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71475::DEBUG::2014-09-25 17:57:03,472::task::993::Storage.TaskManager.Task::(_decref) Task=`ac43239e-c175-4353-9ea6-2d1ea59dea66`::ref 0 aborting False >Thread-71475::DEBUG::2014-09-25 17:57:03,472::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7238320128', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71475::DEBUG::2014-09-25 17:57:03,473::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:03,598::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:03,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134078 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:04,451::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:04,453::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71476::DEBUG::2014-09-25 17:57:04,455::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:57:05,439::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:05,440::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:05,440::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:05,440::__init__::232::IOProcess::(_processLogs) (109968) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:05,441::__init__::232::IOProcess::(_processLogs) (109968) Queuing response >Thread-81::DEBUG::2014-09-25 17:57:05,442::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:57:05,451::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000393482 s, 943 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:05,453::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:05,453::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:05,454::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:05,455::__init__::232::IOProcess::(_processLogs) (109969) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:05,455::__init__::232::IOProcess::(_processLogs) (109969) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:05,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:05,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162744 s, 62.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:07,469::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:07,470::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71477::DEBUG::2014-09-25 17:57:07,472::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:57:07,503::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:07,504::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:07,504::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:07,504::__init__::232::IOProcess::(_processLogs) (109970) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:07,505::__init__::232::IOProcess::(_processLogs) (109970) Queuing response >Thread-80::DEBUG::2014-09-25 17:57:07,506::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:57:07,515::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000616201 s, 613 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:07,516::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:07,516::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:07,517::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:07,517::__init__::232::IOProcess::(_processLogs) (109971) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:07,517::__init__::232::IOProcess::(_processLogs) (109971) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:07,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:07,706::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135208 s, 75.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:57:09,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:09,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135468 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:10,486::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:10,488::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71478::DEBUG::2014-09-25 17:57:10,490::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:57:11,367::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:11,367::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:11,367::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:11,367::__init__::232::IOProcess::(_processLogs) (109972) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:11,368::__init__::232::IOProcess::(_processLogs) (109972) Queuing response >Thread-79::DEBUG::2014-09-25 17:57:11,369::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:57:11,378::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000434197 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:11,379::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:11,380::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:11,380::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:11,380::__init__::232::IOProcess::(_processLogs) (109973) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:11,381::__init__::232::IOProcess::(_processLogs) (109973) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:11,382::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:11,382::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:11,383::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:11,383::__init__::232::IOProcess::(_processLogs) (109974) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:11,383::__init__::232::IOProcess::(_processLogs) (109974) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:11,384::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:11,384::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:11,384::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:11,385::__init__::232::IOProcess::(_processLogs) (109975) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:11,385::__init__::232::IOProcess::(_processLogs) (109975) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:11,386::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:11,386::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:11,386::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:11,387::__init__::232::IOProcess::(_processLogs) (109976) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:11,388::__init__::232::IOProcess::(_processLogs) (109976) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:11,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:11,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135708 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:13,496::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71479::DEBUG::2014-09-25 17:57:13,497::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:57:13,497::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71479::DEBUG::2014-09-25 17:57:13,500::task::595::Storage.TaskManager.Task::(_updateState) Task=`03a2bfa8-4714-493b-be80-baa8aff55926`::moving from state init -> state preparing >Thread-71479::INFO::2014-09-25 17:57:13,500::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71479::INFO::2014-09-25 17:57:13,501::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71479::DEBUG::2014-09-25 17:57:13,501::task::1191::Storage.TaskManager.Task::(prepare) Task=`03a2bfa8-4714-493b-be80-baa8aff55926`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71479::DEBUG::2014-09-25 17:57:13,501::task::595::Storage.TaskManager.Task::(_updateState) Task=`03a2bfa8-4714-493b-be80-baa8aff55926`::moving from state preparing -> state finished >Thread-71479::DEBUG::2014-09-25 17:57:13,501::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71479::DEBUG::2014-09-25 17:57:13,502::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71479::DEBUG::2014-09-25 17:57:13,502::task::993::Storage.TaskManager.Task::(_decref) Task=`03a2bfa8-4714-493b-be80-baa8aff55926`::ref 0 aborting False >Thread-71479::DEBUG::2014-09-25 17:57:13,502::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71479::DEBUG::2014-09-25 17:57:13,502::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:13,504::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:13,505::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71480::DEBUG::2014-09-25 17:57:13,507::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:13,515::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:13,516::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71481::DEBUG::2014-09-25 17:57:13,516::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71481::DEBUG::2014-09-25 17:57:13,518::task::595::Storage.TaskManager.Task::(_updateState) Task=`7b2827bb-c6d7-4ecd-ac78-7990fdacc298`::moving from state init -> state preparing >Thread-71481::INFO::2014-09-25 17:57:13,518::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71481::DEBUG::2014-09-25 17:57:13,519::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`87664ac2-170e-4655-8651-d2c963bc2be8`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71481::DEBUG::2014-09-25 17:57:13,519::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71481::DEBUG::2014-09-25 17:57:13,520::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71481::DEBUG::2014-09-25 17:57:13,520::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`87664ac2-170e-4655-8651-d2c963bc2be8`::Granted request >Thread-71481::DEBUG::2014-09-25 17:57:13,521::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7b2827bb-c6d7-4ecd-ac78-7990fdacc298`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71481::DEBUG::2014-09-25 17:57:13,521::task::993::Storage.TaskManager.Task::(_decref) Task=`7b2827bb-c6d7-4ecd-ac78-7990fdacc298`::ref 1 aborting False >Thread-71481::INFO::2014-09-25 17:57:13,525::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7238320128', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71481::DEBUG::2014-09-25 17:57:13,526::task::1191::Storage.TaskManager.Task::(prepare) Task=`7b2827bb-c6d7-4ecd-ac78-7990fdacc298`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7238320128', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71481::DEBUG::2014-09-25 17:57:13,526::task::595::Storage.TaskManager.Task::(_updateState) Task=`7b2827bb-c6d7-4ecd-ac78-7990fdacc298`::moving from state preparing -> state finished >Thread-71481::DEBUG::2014-09-25 17:57:13,526::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71481::DEBUG::2014-09-25 17:57:13,526::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71481::DEBUG::2014-09-25 17:57:13,527::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71481::DEBUG::2014-09-25 17:57:13,527::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71481::DEBUG::2014-09-25 17:57:13,527::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71481::DEBUG::2014-09-25 17:57:13,527::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71481::DEBUG::2014-09-25 17:57:13,528::task::993::Storage.TaskManager.Task::(_decref) Task=`7b2827bb-c6d7-4ecd-ac78-7990fdacc298`::ref 0 aborting False >Thread-71481::DEBUG::2014-09-25 17:57:13,528::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7238320128', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71481::DEBUG::2014-09-25 17:57:13,529::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:13,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:13,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134122 s, 76.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:15,455::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:15,456::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:15,456::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:15,456::__init__::232::IOProcess::(_processLogs) (109977) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:15,457::__init__::232::IOProcess::(_processLogs) (109977) Queuing response >Thread-81::DEBUG::2014-09-25 17:57:15,458::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:57:15,468::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000435342 s, 852 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:15,470::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:15,470::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:15,470::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:15,471::__init__::232::IOProcess::(_processLogs) (109978) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:15,471::__init__::232::IOProcess::(_processLogs) (109978) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:15,852::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:15,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013405 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:16,522::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:16,524::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71482::DEBUG::2014-09-25 17:57:16,530::task::595::Storage.TaskManager.Task::(_updateState) Task=`55bf6939-f024-4317-aabe-508dc76983ce`::moving from state init -> state preparing >Thread-71482::INFO::2014-09-25 17:57:16,531::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71482::INFO::2014-09-25 17:57:16,532::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000434197', 'lastCheck': '5.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000616201', 'lastCheck': '9.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000435342', 'lastCheck': '1.1', 'valid': True}} >Thread-71482::DEBUG::2014-09-25 17:57:16,533::task::1191::Storage.TaskManager.Task::(prepare) Task=`55bf6939-f024-4317-aabe-508dc76983ce`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000434197', 'lastCheck': '5.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000616201', 'lastCheck': '9.0', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000435342', 'lastCheck': '1.1', 'valid': True}} >Thread-71482::DEBUG::2014-09-25 17:57:16,534::task::595::Storage.TaskManager.Task::(_updateState) Task=`55bf6939-f024-4317-aabe-508dc76983ce`::moving from state preparing -> state finished >Thread-71482::DEBUG::2014-09-25 17:57:16,534::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71482::DEBUG::2014-09-25 17:57:16,535::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71482::DEBUG::2014-09-25 17:57:16,536::task::993::Storage.TaskManager.Task::(_decref) Task=`55bf6939-f024-4317-aabe-508dc76983ce`::ref 0 aborting False >Thread-71482::DEBUG::2014-09-25 17:57:16,541::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:16,562::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:16,565::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71483::DEBUG::2014-09-25 17:57:16,570::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:57:17,520::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:17,520::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:17,520::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:17,520::__init__::232::IOProcess::(_processLogs) (109979) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:17,521::__init__::232::IOProcess::(_processLogs) (109979) Queuing response >Thread-80::DEBUG::2014-09-25 17:57:17,522::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:57:17,531::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000369211 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:17,532::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:17,532::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:17,533::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:17,533::__init__::232::IOProcess::(_processLogs) (109980) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:17,533::__init__::232::IOProcess::(_processLogs) (109980) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:17,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:17,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143548 s, 71.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:19,621::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:19,622::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71484::DEBUG::2014-09-25 17:57:19,624::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:19,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:19,963::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162213 s, 63.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:21,388::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:21,388::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:21,389::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:21,389::__init__::232::IOProcess::(_processLogs) (109981) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:21,389::__init__::232::IOProcess::(_processLogs) (109981) Queuing response >Thread-79::DEBUG::2014-09-25 17:57:21,391::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:57:21,399::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000310153 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:21,400::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:21,401::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:21,401::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:21,401::__init__::232::IOProcess::(_processLogs) (109982) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:21,402::__init__::232::IOProcess::(_processLogs) (109982) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:21,403::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:21,404::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:21,404::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:21,404::__init__::232::IOProcess::(_processLogs) (109983) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:21,405::__init__::232::IOProcess::(_processLogs) (109983) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:21,405::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:21,406::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:21,406::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:21,406::__init__::232::IOProcess::(_processLogs) (109984) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:21,407::__init__::232::IOProcess::(_processLogs) (109984) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:21,407::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:21,408::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:21,408::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:21,409::__init__::232::IOProcess::(_processLogs) (109985) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:21,409::__init__::232::IOProcess::(_processLogs) (109985) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:21,983::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:22,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134013 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:22,636::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:22,637::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71485::DEBUG::2014-09-25 17:57:22,639::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:23,551::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:23,553::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71486::DEBUG::2014-09-25 17:57:23,553::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71486::DEBUG::2014-09-25 17:57:23,555::task::595::Storage.TaskManager.Task::(_updateState) Task=`4bf66e29-0fae-4e8f-acd9-962f70e72009`::moving from state init -> state preparing >Thread-71486::INFO::2014-09-25 17:57:23,555::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71486::INFO::2014-09-25 17:57:23,556::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71486::DEBUG::2014-09-25 17:57:23,556::task::1191::Storage.TaskManager.Task::(prepare) Task=`4bf66e29-0fae-4e8f-acd9-962f70e72009`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71486::DEBUG::2014-09-25 17:57:23,556::task::595::Storage.TaskManager.Task::(_updateState) Task=`4bf66e29-0fae-4e8f-acd9-962f70e72009`::moving from state preparing -> state finished >Thread-71486::DEBUG::2014-09-25 17:57:23,557::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71486::DEBUG::2014-09-25 17:57:23,557::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71486::DEBUG::2014-09-25 17:57:23,557::task::993::Storage.TaskManager.Task::(_decref) Task=`4bf66e29-0fae-4e8f-acd9-962f70e72009`::ref 0 aborting False >Thread-71486::DEBUG::2014-09-25 17:57:23,557::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71486::DEBUG::2014-09-25 17:57:23,558::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:23,569::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71487::DEBUG::2014-09-25 17:57:23,570::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:57:23,570::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71487::DEBUG::2014-09-25 17:57:23,572::task::595::Storage.TaskManager.Task::(_updateState) Task=`34bab4ef-0d20-4d25-8404-640e83d5710d`::moving from state init -> state preparing >Thread-71487::INFO::2014-09-25 17:57:23,572::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71487::DEBUG::2014-09-25 17:57:23,573::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6e25b2d1-80ce-4a8d-9823-a754e3a9bc79`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71487::DEBUG::2014-09-25 17:57:23,573::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71487::DEBUG::2014-09-25 17:57:23,574::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71487::DEBUG::2014-09-25 17:57:23,574::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6e25b2d1-80ce-4a8d-9823-a754e3a9bc79`::Granted request >Thread-71487::DEBUG::2014-09-25 17:57:23,575::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`34bab4ef-0d20-4d25-8404-640e83d5710d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71487::DEBUG::2014-09-25 17:57:23,575::task::993::Storage.TaskManager.Task::(_decref) Task=`34bab4ef-0d20-4d25-8404-640e83d5710d`::ref 1 aborting False >Thread-71487::INFO::2014-09-25 17:57:23,579::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7238320128', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71487::DEBUG::2014-09-25 17:57:23,579::task::1191::Storage.TaskManager.Task::(prepare) Task=`34bab4ef-0d20-4d25-8404-640e83d5710d`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7238320128', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71487::DEBUG::2014-09-25 17:57:23,580::task::595::Storage.TaskManager.Task::(_updateState) Task=`34bab4ef-0d20-4d25-8404-640e83d5710d`::moving from state preparing -> state finished >Thread-71487::DEBUG::2014-09-25 17:57:23,580::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71487::DEBUG::2014-09-25 17:57:23,580::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71487::DEBUG::2014-09-25 17:57:23,580::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71487::DEBUG::2014-09-25 17:57:23,581::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71487::DEBUG::2014-09-25 17:57:23,581::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71487::DEBUG::2014-09-25 17:57:23,581::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71487::DEBUG::2014-09-25 17:57:23,582::task::993::Storage.TaskManager.Task::(_decref) Task=`34bab4ef-0d20-4d25-8404-640e83d5710d`::ref 0 aborting False >Thread-71487::DEBUG::2014-09-25 17:57:23,582::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7238320128', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71487::DEBUG::2014-09-25 17:57:23,582::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:24,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:24,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185386 s, 55.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:25,474::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:25,474::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:25,475::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:57:25,476::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:57:25,476::__init__::232::IOProcess::(_processLogs) (109986) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:25,484::__init__::232::IOProcess::(_processLogs) (109986) Queuing response >Thread-81::DEBUG::2014-09-25 17:57:25,486::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000458038 s, 810 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:25,487::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:25,487::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:25,487::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:25,487::__init__::232::IOProcess::(_processLogs) (109987) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:25,488::__init__::232::IOProcess::(_processLogs) (109987) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:25,652::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:25,653::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71488::DEBUG::2014-09-25 17:57:25,656::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:26,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:26,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0125779 s, 81.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:27,536::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:27,536::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:27,536::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:27,537::__init__::232::IOProcess::(_processLogs) (109988) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:27,537::__init__::232::IOProcess::(_processLogs) (109988) Queuing response >Thread-80::DEBUG::2014-09-25 17:57:27,538::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:57:27,547::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00062068 s, 609 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:27,548::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:27,549::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:27,549::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:27,549::__init__::232::IOProcess::(_processLogs) (109989) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:27,550::__init__::232::IOProcess::(_processLogs) (109989) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:28,115::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:28,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149361 s, 68.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:28,670::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:28,671::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71489::DEBUG::2014-09-25 17:57:28,673::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:30,158::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:30,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133962 s, 76.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:31,409::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:31,410::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:31,410::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:31,410::__init__::232::IOProcess::(_processLogs) (109990) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:31,411::__init__::232::IOProcess::(_processLogs) (109990) Queuing response >Thread-79::DEBUG::2014-09-25 17:57:31,412::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:57:31,421::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000416793 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:31,422::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:31,422::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:31,423::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:31,423::__init__::232::IOProcess::(_processLogs) (109991) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:31,423::__init__::232::IOProcess::(_processLogs) (109991) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:31,425::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:31,425::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:31,426::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:31,426::__init__::232::IOProcess::(_processLogs) (109992) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:31,427::__init__::232::IOProcess::(_processLogs) (109992) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:31,427::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:31,427::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:31,428::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:31,428::__init__::232::IOProcess::(_processLogs) (109993) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:31,428::__init__::232::IOProcess::(_processLogs) (109993) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:31,429::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:31,429::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:31,430::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:31,431::__init__::232::IOProcess::(_processLogs) (109994) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:31,431::__init__::232::IOProcess::(_processLogs) (109994) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:31,685::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:31,687::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71490::DEBUG::2014-09-25 17:57:31,689::task::595::Storage.TaskManager.Task::(_updateState) Task=`4107b7bd-4bd1-4f00-9757-3bfc6d7e0db1`::moving from state init -> state preparing >Thread-71490::INFO::2014-09-25 17:57:31,690::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71490::INFO::2014-09-25 17:57:31,690::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000416793', 'lastCheck': '0.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00062068', 'lastCheck': '4.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000458038', 'lastCheck': '6.2', 'valid': True}} >Thread-71490::DEBUG::2014-09-25 17:57:31,690::task::1191::Storage.TaskManager.Task::(prepare) Task=`4107b7bd-4bd1-4f00-9757-3bfc6d7e0db1`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000416793', 'lastCheck': '0.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00062068', 'lastCheck': '4.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000458038', 'lastCheck': '6.2', 'valid': True}} >Thread-71490::DEBUG::2014-09-25 17:57:31,691::task::595::Storage.TaskManager.Task::(_updateState) Task=`4107b7bd-4bd1-4f00-9757-3bfc6d7e0db1`::moving from state preparing -> state finished >Thread-71490::DEBUG::2014-09-25 17:57:31,691::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71490::DEBUG::2014-09-25 17:57:31,691::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71490::DEBUG::2014-09-25 17:57:31,691::task::993::Storage.TaskManager.Task::(_decref) Task=`4107b7bd-4bd1-4f00-9757-3bfc6d7e0db1`::ref 0 aborting False >Thread-71490::DEBUG::2014-09-25 17:57:31,694::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:31,708::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:31,709::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71491::DEBUG::2014-09-25 17:57:31,711::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:32,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:32,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013543 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:33,608::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71492::DEBUG::2014-09-25 17:57:33,610::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:57:33,610::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71492::DEBUG::2014-09-25 17:57:33,612::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5bd7019-3d9b-4519-8195-35511d150679`::moving from state init -> state preparing >Thread-71492::INFO::2014-09-25 17:57:33,612::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71492::INFO::2014-09-25 17:57:33,613::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71492::DEBUG::2014-09-25 17:57:33,613::task::1191::Storage.TaskManager.Task::(prepare) Task=`a5bd7019-3d9b-4519-8195-35511d150679`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71492::DEBUG::2014-09-25 17:57:33,614::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5bd7019-3d9b-4519-8195-35511d150679`::moving from state preparing -> state finished >Thread-71492::DEBUG::2014-09-25 17:57:33,614::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71492::DEBUG::2014-09-25 17:57:33,614::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71492::DEBUG::2014-09-25 17:57:33,614::task::993::Storage.TaskManager.Task::(_decref) Task=`a5bd7019-3d9b-4519-8195-35511d150679`::ref 0 aborting False >Thread-71492::DEBUG::2014-09-25 17:57:33,615::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71492::DEBUG::2014-09-25 17:57:33,615::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:33,625::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71493::DEBUG::2014-09-25 17:57:33,626::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:57:33,627::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71493::DEBUG::2014-09-25 17:57:33,629::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef1ede2e-1e4e-427b-b4d3-7c1872ff8882`::moving from state init -> state preparing >Thread-71493::INFO::2014-09-25 17:57:33,629::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71493::DEBUG::2014-09-25 17:57:33,630::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f78da76f-45b3-423d-96ed-a9db6021c3a5`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71493::DEBUG::2014-09-25 17:57:33,630::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71493::DEBUG::2014-09-25 17:57:33,630::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71493::DEBUG::2014-09-25 17:57:33,630::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f78da76f-45b3-423d-96ed-a9db6021c3a5`::Granted request >Thread-71493::DEBUG::2014-09-25 17:57:33,631::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ef1ede2e-1e4e-427b-b4d3-7c1872ff8882`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71493::DEBUG::2014-09-25 17:57:33,631::task::993::Storage.TaskManager.Task::(_decref) Task=`ef1ede2e-1e4e-427b-b4d3-7c1872ff8882`::ref 1 aborting False >Thread-71493::INFO::2014-09-25 17:57:33,636::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71493::DEBUG::2014-09-25 17:57:33,636::task::1191::Storage.TaskManager.Task::(prepare) Task=`ef1ede2e-1e4e-427b-b4d3-7c1872ff8882`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71493::DEBUG::2014-09-25 17:57:33,636::task::595::Storage.TaskManager.Task::(_updateState) Task=`ef1ede2e-1e4e-427b-b4d3-7c1872ff8882`::moving from state preparing -> state finished >Thread-71493::DEBUG::2014-09-25 17:57:33,636::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71493::DEBUG::2014-09-25 17:57:33,637::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71493::DEBUG::2014-09-25 17:57:33,637::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71493::DEBUG::2014-09-25 17:57:33,637::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71493::DEBUG::2014-09-25 17:57:33,638::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71493::DEBUG::2014-09-25 17:57:33,638::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71493::DEBUG::2014-09-25 17:57:33,638::task::993::Storage.TaskManager.Task::(_decref) Task=`ef1ede2e-1e4e-427b-b4d3-7c1872ff8882`::ref 0 aborting False >Thread-71493::DEBUG::2014-09-25 17:57:33,639::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71493::DEBUG::2014-09-25 17:57:33,639::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:34,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:34,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132344 s, 77.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:34,755::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:34,757::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71494::DEBUG::2014-09-25 17:57:34,759::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 17:57:35,490::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:57:35,494::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:35,494::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:35,495::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:35,495::__init__::232::IOProcess::(_processLogs) (109995) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:57:35,495::__init__::232::IOProcess::(_processLogs) (109995) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:35,496::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:35,497::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:35,499::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:35,500::__init__::232::IOProcess::(_processLogs) (109996) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:57:35,503::__init__::232::IOProcess::(_processLogs) (109996) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:35,506::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:35,508::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:35,509::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:35,510::__init__::232::IOProcess::(_processLogs) (109997) Got request for method 'glob' >Thread-81::DEBUG::2014-09-25 17:57:35,510::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 17:57:35,512::__init__::232::IOProcess::(_processLogs) (109997) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:35,513::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:35,513::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:35,514::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:35,514::__init__::232::IOProcess::(_processLogs) (109998) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:57:35,582::__init__::232::IOProcess::(_processLogs) (109998) Queuing response >Thread-81::DEBUG::2014-09-25 17:57:35,583::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:57:35,584::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:35,584::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:35,584::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:35,585::__init__::232::IOProcess::(_processLogs) (109999) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:57:35,585::__init__::232::IOProcess::(_processLogs) (109999) Queuing response >Thread-81::DEBUG::2014-09-25 17:57:35,585::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 17:57:35,587::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:35,587::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:35,588::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:35,588::__init__::232::IOProcess::(_processLogs) (110000) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:35,588::__init__::232::IOProcess::(_processLogs) (110000) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:35,589::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:35,589::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:35,589::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:35,590::__init__::232::IOProcess::(_processLogs) (110001) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:57:35,590::__init__::232::IOProcess::(_processLogs) (110001) Queuing response >Thread-81::DEBUG::2014-09-25 17:57:35,590::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 17:57:35,591::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 17:57:35,591::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:57:35,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:35,592::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:35,592::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:35,593::__init__::232::IOProcess::(_processLogs) (110002) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:35,593::__init__::232::IOProcess::(_processLogs) (110002) Queuing response >Thread-81::DEBUG::2014-09-25 17:57:35,594::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:57:35,603::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000395946 s, 937 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:35,604::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:35,604::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:35,605::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:35,605::__init__::232::IOProcess::(_processLogs) (110003) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:35,606::__init__::232::IOProcess::(_processLogs) (110003) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:36,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:36,306::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134421 s, 76.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:37,552::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:37,552::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:37,553::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:37,553::__init__::232::IOProcess::(_processLogs) (110004) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:37,553::__init__::232::IOProcess::(_processLogs) (110004) Queuing response >Thread-80::DEBUG::2014-09-25 17:57:37,555::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:57:37,564::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000533201 s, 709 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:37,565::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:37,565::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:37,565::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:37,566::__init__::232::IOProcess::(_processLogs) (110005) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:37,566::__init__::232::IOProcess::(_processLogs) (110005) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:37,771::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:37,773::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71498::DEBUG::2014-09-25 17:57:37,775::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:38,326::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:38,348::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130505 s, 78.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:57:40,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:40,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137823 s, 74.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:40,789::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:40,791::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71499::DEBUG::2014-09-25 17:57:40,814::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:57:41,431::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:41,432::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:41,432::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:57:41,433::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:57:41,434::__init__::232::IOProcess::(_processLogs) (110006) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:41,442::__init__::232::IOProcess::(_processLogs) (110006) Queuing response >Thread-79::DEBUG::2014-09-25 17:57:41,443::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000518059 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:41,444::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:41,444::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:41,444::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:41,445::__init__::232::IOProcess::(_processLogs) (110007) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:41,445::__init__::232::IOProcess::(_processLogs) (110007) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:41,447::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:41,447::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:41,447::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:41,448::__init__::232::IOProcess::(_processLogs) (110008) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:41,448::__init__::232::IOProcess::(_processLogs) (110008) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:41,449::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:41,449::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:41,450::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:41,450::__init__::232::IOProcess::(_processLogs) (110009) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:41,450::__init__::232::IOProcess::(_processLogs) (110009) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:41,451::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:41,451::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:41,452::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:41,452::__init__::232::IOProcess::(_processLogs) (110010) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:41,453::__init__::232::IOProcess::(_processLogs) (110010) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:42,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:42,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013364 s, 76.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:43,665::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71500::DEBUG::2014-09-25 17:57:43,667::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:57:43,667::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71500::DEBUG::2014-09-25 17:57:43,669::task::595::Storage.TaskManager.Task::(_updateState) Task=`cbfcf5b0-39e0-4add-b1c6-a86455737963`::moving from state init -> state preparing >Thread-71500::INFO::2014-09-25 17:57:43,670::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71500::INFO::2014-09-25 17:57:43,670::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71500::DEBUG::2014-09-25 17:57:43,671::task::1191::Storage.TaskManager.Task::(prepare) Task=`cbfcf5b0-39e0-4add-b1c6-a86455737963`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71500::DEBUG::2014-09-25 17:57:43,671::task::595::Storage.TaskManager.Task::(_updateState) Task=`cbfcf5b0-39e0-4add-b1c6-a86455737963`::moving from state preparing -> state finished >Thread-71500::DEBUG::2014-09-25 17:57:43,671::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71500::DEBUG::2014-09-25 17:57:43,671::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71500::DEBUG::2014-09-25 17:57:43,671::task::993::Storage.TaskManager.Task::(_decref) Task=`cbfcf5b0-39e0-4add-b1c6-a86455737963`::ref 0 aborting False >Thread-71500::DEBUG::2014-09-25 17:57:43,672::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71500::DEBUG::2014-09-25 17:57:43,672::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:43,686::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:43,687::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71501::DEBUG::2014-09-25 17:57:43,687::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71501::DEBUG::2014-09-25 17:57:43,689::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ed9943d-de00-4d23-b3d0-bb5705abb9db`::moving from state init -> state preparing >Thread-71501::INFO::2014-09-25 17:57:43,689::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71501::DEBUG::2014-09-25 17:57:43,690::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a9accbe8-b0f1-4ae2-b829-5292ea02cb50`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71501::DEBUG::2014-09-25 17:57:43,690::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71501::DEBUG::2014-09-25 17:57:43,690::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71501::DEBUG::2014-09-25 17:57:43,691::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a9accbe8-b0f1-4ae2-b829-5292ea02cb50`::Granted request >Thread-71501::DEBUG::2014-09-25 17:57:43,691::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4ed9943d-de00-4d23-b3d0-bb5705abb9db`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71501::DEBUG::2014-09-25 17:57:43,692::task::993::Storage.TaskManager.Task::(_decref) Task=`4ed9943d-de00-4d23-b3d0-bb5705abb9db`::ref 1 aborting False >Thread-71501::INFO::2014-09-25 17:57:43,696::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71501::DEBUG::2014-09-25 17:57:43,696::task::1191::Storage.TaskManager.Task::(prepare) Task=`4ed9943d-de00-4d23-b3d0-bb5705abb9db`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71501::DEBUG::2014-09-25 17:57:43,697::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ed9943d-de00-4d23-b3d0-bb5705abb9db`::moving from state preparing -> state finished >Thread-71501::DEBUG::2014-09-25 17:57:43,697::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71501::DEBUG::2014-09-25 17:57:43,697::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71501::DEBUG::2014-09-25 17:57:43,697::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71501::DEBUG::2014-09-25 17:57:43,698::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71501::DEBUG::2014-09-25 17:57:43,698::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71501::DEBUG::2014-09-25 17:57:43,698::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71501::DEBUG::2014-09-25 17:57:43,698::task::993::Storage.TaskManager.Task::(_decref) Task=`4ed9943d-de00-4d23-b3d0-bb5705abb9db`::ref 0 aborting False >Thread-71501::DEBUG::2014-09-25 17:57:43,699::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71501::DEBUG::2014-09-25 17:57:43,699::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:43,831::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:43,833::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71502::DEBUG::2014-09-25 17:57:43,835::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:44,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:44,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134712 s, 76.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:45,608::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:45,608::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:45,608::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:45,609::__init__::232::IOProcess::(_processLogs) (110011) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:45,609::__init__::232::IOProcess::(_processLogs) (110011) Queuing response >Thread-81::DEBUG::2014-09-25 17:57:45,611::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:57:45,620::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000401635 s, 924 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:45,621::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:45,621::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:45,621::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:45,622::__init__::232::IOProcess::(_processLogs) (110012) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:45,622::__init__::232::IOProcess::(_processLogs) (110012) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:46,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:46,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0224151 s, 45.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:46,851::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:46,854::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71503::DEBUG::2014-09-25 17:57:46,862::task::595::Storage.TaskManager.Task::(_updateState) Task=`20c74679-88f7-4d6a-9851-eaf687b6e493`::moving from state init -> state preparing >Thread-71503::INFO::2014-09-25 17:57:46,863::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71503::INFO::2014-09-25 17:57:46,865::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000518059', 'lastCheck': '5.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000533201', 'lastCheck': '9.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000401635', 'lastCheck': '1.2', 'valid': True}} >Thread-71503::DEBUG::2014-09-25 17:57:46,866::task::1191::Storage.TaskManager.Task::(prepare) Task=`20c74679-88f7-4d6a-9851-eaf687b6e493`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000518059', 'lastCheck': '5.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000533201', 'lastCheck': '9.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000401635', 'lastCheck': '1.2', 'valid': True}} >Thread-71503::DEBUG::2014-09-25 17:57:46,867::task::595::Storage.TaskManager.Task::(_updateState) Task=`20c74679-88f7-4d6a-9851-eaf687b6e493`::moving from state preparing -> state finished >Thread-71503::DEBUG::2014-09-25 17:57:46,868::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71503::DEBUG::2014-09-25 17:57:46,869::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71503::DEBUG::2014-09-25 17:57:46,870::task::993::Storage.TaskManager.Task::(_decref) Task=`20c74679-88f7-4d6a-9851-eaf687b6e493`::ref 0 aborting False >Thread-71503::DEBUG::2014-09-25 17:57:46,878::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:46,904::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:46,910::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71504::DEBUG::2014-09-25 17:57:46,922::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:57:47,569::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:47,569::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:47,569::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:47,569::__init__::232::IOProcess::(_processLogs) (110013) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:47,571::__init__::232::IOProcess::(_processLogs) (110013) Queuing response >Thread-80::DEBUG::2014-09-25 17:57:47,572::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:57:47,581::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000411156 s, 919 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:47,583::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:47,583::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:47,583::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:47,584::__init__::232::IOProcess::(_processLogs) (110014) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:47,584::__init__::232::IOProcess::(_processLogs) (110014) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:48,717::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:48,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147954 s, 69.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:49,988::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:49,989::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71505::DEBUG::2014-09-25 17:57:49,991::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:50,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:50,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136996 s, 74.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:51,453::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:51,453::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:51,454::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:57:51,455::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:57:51,455::__init__::232::IOProcess::(_processLogs) (110015) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:51,464::__init__::232::IOProcess::(_processLogs) (110015) Queuing response >Thread-79::DEBUG::2014-09-25 17:57:51,465::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000444124 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:51,466::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:51,466::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:51,466::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:51,467::__init__::232::IOProcess::(_processLogs) (110016) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:51,467::__init__::232::IOProcess::(_processLogs) (110016) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:51,469::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:51,469::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:51,469::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:51,469::__init__::232::IOProcess::(_processLogs) (110017) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:51,470::__init__::232::IOProcess::(_processLogs) (110017) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:51,471::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:51,471::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:51,471::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:51,471::__init__::232::IOProcess::(_processLogs) (110018) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:51,472::__init__::232::IOProcess::(_processLogs) (110018) Queuing response >Thread-29::DEBUG::2014-09-25 17:57:51,473::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:51,473::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:51,473::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:51,474::__init__::232::IOProcess::(_processLogs) (110019) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:57:51,474::__init__::232::IOProcess::(_processLogs) (110019) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:52,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:52,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134307 s, 76.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:53,008::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:53,009::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71506::DEBUG::2014-09-25 17:57:53,011::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:53,731::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:53,733::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71507::DEBUG::2014-09-25 17:57:53,733::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71507::DEBUG::2014-09-25 17:57:53,735::task::595::Storage.TaskManager.Task::(_updateState) Task=`73f916f5-a9bc-42fb-a1a2-f8d104774db0`::moving from state init -> state preparing >Thread-71507::INFO::2014-09-25 17:57:53,735::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71507::INFO::2014-09-25 17:57:53,736::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71507::DEBUG::2014-09-25 17:57:53,736::task::1191::Storage.TaskManager.Task::(prepare) Task=`73f916f5-a9bc-42fb-a1a2-f8d104774db0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71507::DEBUG::2014-09-25 17:57:53,736::task::595::Storage.TaskManager.Task::(_updateState) Task=`73f916f5-a9bc-42fb-a1a2-f8d104774db0`::moving from state preparing -> state finished >Thread-71507::DEBUG::2014-09-25 17:57:53,737::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71507::DEBUG::2014-09-25 17:57:53,737::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71507::DEBUG::2014-09-25 17:57:53,737::task::993::Storage.TaskManager.Task::(_decref) Task=`73f916f5-a9bc-42fb-a1a2-f8d104774db0`::ref 0 aborting False >Thread-71507::DEBUG::2014-09-25 17:57:53,737::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71507::DEBUG::2014-09-25 17:57:53,738::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:53,752::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:53,753::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71508::DEBUG::2014-09-25 17:57:53,753::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71508::DEBUG::2014-09-25 17:57:53,755::task::595::Storage.TaskManager.Task::(_updateState) Task=`68b73d4e-dc43-4205-bf44-2723b0f3d06a`::moving from state init -> state preparing >Thread-71508::INFO::2014-09-25 17:57:53,755::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71508::DEBUG::2014-09-25 17:57:53,756::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`75d7fa37-7abb-4085-955f-529a5bff071b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71508::DEBUG::2014-09-25 17:57:53,756::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71508::DEBUG::2014-09-25 17:57:53,756::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71508::DEBUG::2014-09-25 17:57:53,757::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`75d7fa37-7abb-4085-955f-529a5bff071b`::Granted request >Thread-71508::DEBUG::2014-09-25 17:57:53,757::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`68b73d4e-dc43-4205-bf44-2723b0f3d06a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71508::DEBUG::2014-09-25 17:57:53,757::task::993::Storage.TaskManager.Task::(_decref) Task=`68b73d4e-dc43-4205-bf44-2723b0f3d06a`::ref 1 aborting False >Thread-71508::INFO::2014-09-25 17:57:53,762::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71508::DEBUG::2014-09-25 17:57:53,762::task::1191::Storage.TaskManager.Task::(prepare) Task=`68b73d4e-dc43-4205-bf44-2723b0f3d06a`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71508::DEBUG::2014-09-25 17:57:53,762::task::595::Storage.TaskManager.Task::(_updateState) Task=`68b73d4e-dc43-4205-bf44-2723b0f3d06a`::moving from state preparing -> state finished >Thread-71508::DEBUG::2014-09-25 17:57:53,762::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71508::DEBUG::2014-09-25 17:57:53,763::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71508::DEBUG::2014-09-25 17:57:53,763::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71508::DEBUG::2014-09-25 17:57:53,763::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71508::DEBUG::2014-09-25 17:57:53,764::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71508::DEBUG::2014-09-25 17:57:53,764::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71508::DEBUG::2014-09-25 17:57:53,764::task::993::Storage.TaskManager.Task::(_decref) Task=`68b73d4e-dc43-4205-bf44-2723b0f3d06a`::ref 0 aborting False >Thread-71508::DEBUG::2014-09-25 17:57:53,765::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71508::DEBUG::2014-09-25 17:57:53,765::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:54,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:54,867::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013694 s, 74.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:55,625::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:55,625::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:55,625::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:55,626::__init__::232::IOProcess::(_processLogs) (110020) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:55,626::__init__::232::IOProcess::(_processLogs) (110020) Queuing response >Thread-81::DEBUG::2014-09-25 17:57:55,627::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:57:55,636::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000498434 s, 744 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:55,637::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:55,638::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:55,638::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:55,638::__init__::232::IOProcess::(_processLogs) (110021) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:55,639::__init__::232::IOProcess::(_processLogs) (110021) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:56,030::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:56,031::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71509::DEBUG::2014-09-25 17:57:56,033::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:57:56,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:56,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137862 s, 74.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:57,586::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:57,587::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:57,587::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:57,587::__init__::232::IOProcess::(_processLogs) (110022) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:57,588::__init__::232::IOProcess::(_processLogs) (110022) Queuing response >Thread-80::DEBUG::2014-09-25 17:57:57,589::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:57:57,598::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000540432 s, 699 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:57:57,599::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:57:57,599::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:57:57,600::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:57:57,600::__init__::232::IOProcess::(_processLogs) (110023) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:57:57,600::__init__::232::IOProcess::(_processLogs) (110023) Queuing response >Dummy-148::DEBUG::2014-09-25 17:57:58,929::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:57:58,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020679 s, 49.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:57:59,052::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:57:59,053::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71510::DEBUG::2014-09-25 17:57:59,055::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:00,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:01,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131021 s, 78.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:01,476::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:01,476::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:01,477::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:58:01,478::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:58:01,478::__init__::232::IOProcess::(_processLogs) (110024) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:01,487::__init__::232::IOProcess::(_processLogs) (110024) Queuing response >Thread-79::DEBUG::2014-09-25 17:58:01,490::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00118282 s, 475 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:01,490::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:01,491::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:01,491::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:01,491::__init__::232::IOProcess::(_processLogs) (110025) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:01,492::__init__::232::IOProcess::(_processLogs) (110025) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:01,493::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:01,494::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:01,495::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:01,495::__init__::232::IOProcess::(_processLogs) (110026) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:01,495::__init__::232::IOProcess::(_processLogs) (110026) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:01,495::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:01,496::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:01,496::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:01,496::__init__::232::IOProcess::(_processLogs) (110027) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:01,496::__init__::232::IOProcess::(_processLogs) (110027) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:01,497::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:01,498::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:01,499::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:01,499::__init__::232::IOProcess::(_processLogs) (110028) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:01,499::__init__::232::IOProcess::(_processLogs) (110028) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:02,070::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:02,071::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71511::DEBUG::2014-09-25 17:58:02,074::task::595::Storage.TaskManager.Task::(_updateState) Task=`b70791b3-9fd4-44d1-872a-fc90f778c917`::moving from state init -> state preparing >Thread-71511::INFO::2014-09-25 17:58:02,074::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71511::INFO::2014-09-25 17:58:02,075::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00118282', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000540432', 'lastCheck': '4.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000498434', 'lastCheck': '6.4', 'valid': True}} >Thread-71511::DEBUG::2014-09-25 17:58:02,075::task::1191::Storage.TaskManager.Task::(prepare) Task=`b70791b3-9fd4-44d1-872a-fc90f778c917`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00118282', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000540432', 'lastCheck': '4.5', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000498434', 'lastCheck': '6.4', 'valid': True}} >Thread-71511::DEBUG::2014-09-25 17:58:02,075::task::595::Storage.TaskManager.Task::(_updateState) Task=`b70791b3-9fd4-44d1-872a-fc90f778c917`::moving from state preparing -> state finished >Thread-71511::DEBUG::2014-09-25 17:58:02,075::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71511::DEBUG::2014-09-25 17:58:02,076::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71511::DEBUG::2014-09-25 17:58:02,076::task::993::Storage.TaskManager.Task::(_decref) Task=`b70791b3-9fd4-44d1-872a-fc90f778c917`::ref 0 aborting False >Thread-71511::DEBUG::2014-09-25 17:58:02,078::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:02,100::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:02,101::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71512::DEBUG::2014-09-25 17:58:02,103::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:03,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:03,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136305 s, 75.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:03,799::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71513::DEBUG::2014-09-25 17:58:03,800::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:58:03,800::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71513::DEBUG::2014-09-25 17:58:03,803::task::595::Storage.TaskManager.Task::(_updateState) Task=`fa223eed-3702-456f-b7d1-f843ccfe9063`::moving from state init -> state preparing >Thread-71513::INFO::2014-09-25 17:58:03,803::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71513::INFO::2014-09-25 17:58:03,804::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71513::DEBUG::2014-09-25 17:58:03,804::task::1191::Storage.TaskManager.Task::(prepare) Task=`fa223eed-3702-456f-b7d1-f843ccfe9063`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71513::DEBUG::2014-09-25 17:58:03,804::task::595::Storage.TaskManager.Task::(_updateState) Task=`fa223eed-3702-456f-b7d1-f843ccfe9063`::moving from state preparing -> state finished >Thread-71513::DEBUG::2014-09-25 17:58:03,804::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71513::DEBUG::2014-09-25 17:58:03,805::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71513::DEBUG::2014-09-25 17:58:03,805::task::993::Storage.TaskManager.Task::(_decref) Task=`fa223eed-3702-456f-b7d1-f843ccfe9063`::ref 0 aborting False >Thread-71513::DEBUG::2014-09-25 17:58:03,805::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71513::DEBUG::2014-09-25 17:58:03,806::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:03,820::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:03,821::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71514::DEBUG::2014-09-25 17:58:03,821::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71514::DEBUG::2014-09-25 17:58:03,823::task::595::Storage.TaskManager.Task::(_updateState) Task=`6100c030-2e48-4bea-b260-b40fd125fff7`::moving from state init -> state preparing >Thread-71514::INFO::2014-09-25 17:58:03,823::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71514::DEBUG::2014-09-25 17:58:03,824::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6485319f-917d-4fbd-8d7d-4646e87600b0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71514::DEBUG::2014-09-25 17:58:03,824::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71514::DEBUG::2014-09-25 17:58:03,825::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71514::DEBUG::2014-09-25 17:58:03,825::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6485319f-917d-4fbd-8d7d-4646e87600b0`::Granted request >Thread-71514::DEBUG::2014-09-25 17:58:03,826::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`6100c030-2e48-4bea-b260-b40fd125fff7`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71514::DEBUG::2014-09-25 17:58:03,826::task::993::Storage.TaskManager.Task::(_decref) Task=`6100c030-2e48-4bea-b260-b40fd125fff7`::ref 1 aborting False >Thread-71514::INFO::2014-09-25 17:58:03,830::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71514::DEBUG::2014-09-25 17:58:03,830::task::1191::Storage.TaskManager.Task::(prepare) Task=`6100c030-2e48-4bea-b260-b40fd125fff7`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71514::DEBUG::2014-09-25 17:58:03,831::task::595::Storage.TaskManager.Task::(_updateState) Task=`6100c030-2e48-4bea-b260-b40fd125fff7`::moving from state preparing -> state finished >Thread-71514::DEBUG::2014-09-25 17:58:03,831::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71514::DEBUG::2014-09-25 17:58:03,831::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71514::DEBUG::2014-09-25 17:58:03,832::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71514::DEBUG::2014-09-25 17:58:03,832::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71514::DEBUG::2014-09-25 17:58:03,832::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71514::DEBUG::2014-09-25 17:58:03,832::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71514::DEBUG::2014-09-25 17:58:03,833::task::993::Storage.TaskManager.Task::(_decref) Task=`6100c030-2e48-4bea-b260-b40fd125fff7`::ref 0 aborting False >Thread-71514::DEBUG::2014-09-25 17:58:03,833::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71514::DEBUG::2014-09-25 17:58:03,834::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:05,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:05,087::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013397 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:05,184::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:05,185::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71515::DEBUG::2014-09-25 17:58:05,188::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:58:05,641::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:05,641::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:05,641::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:05,642::__init__::232::IOProcess::(_processLogs) (110029) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:05,642::__init__::232::IOProcess::(_processLogs) (110029) Queuing response >Thread-81::DEBUG::2014-09-25 17:58:05,643::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:58:05,652::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000426545 s, 870 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:05,653::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:05,654::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:05,654::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:05,654::__init__::232::IOProcess::(_processLogs) (110030) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:05,655::__init__::232::IOProcess::(_processLogs) (110030) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:07,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:07,144::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0238682 s, 42.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:07,603::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:07,603::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:07,603::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:07,604::__init__::232::IOProcess::(_processLogs) (110031) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:07,604::__init__::232::IOProcess::(_processLogs) (110031) Queuing response >Thread-80::DEBUG::2014-09-25 17:58:07,605::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:58:07,614::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000530981 s, 712 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:07,616::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:07,616::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:07,616::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:07,617::__init__::232::IOProcess::(_processLogs) (110032) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:07,617::__init__::232::IOProcess::(_processLogs) (110032) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:08,204::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:08,205::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71516::DEBUG::2014-09-25 17:58:08,207::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:09,283::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:09,305::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133888 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:11,225::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:11,227::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71517::DEBUG::2014-09-25 17:58:11,229::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:11,325::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:11,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134393 s, 76.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:11,500::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:11,500::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:11,500::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:11,501::__init__::232::IOProcess::(_processLogs) (110033) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:11,502::__init__::232::IOProcess::(_processLogs) (110033) Queuing response >Thread-79::DEBUG::2014-09-25 17:58:11,502::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:58:11,511::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000470116 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:11,513::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:11,513::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:11,513::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:11,513::__init__::232::IOProcess::(_processLogs) (110034) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:11,514::__init__::232::IOProcess::(_processLogs) (110034) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:11,515::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:11,516::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:11,516::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:11,516::__init__::232::IOProcess::(_processLogs) (110035) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:11,516::__init__::232::IOProcess::(_processLogs) (110035) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:11,517::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:11,517::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:11,518::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:11,518::__init__::232::IOProcess::(_processLogs) (110036) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:11,518::__init__::232::IOProcess::(_processLogs) (110036) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:11,519::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:11,519::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:11,520::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:11,520::__init__::232::IOProcess::(_processLogs) (110037) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:11,520::__init__::232::IOProcess::(_processLogs) (110037) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:13,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:13,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134588 s, 76.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:13,862::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:13,864::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71518::DEBUG::2014-09-25 17:58:13,863::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71518::DEBUG::2014-09-25 17:58:13,866::task::595::Storage.TaskManager.Task::(_updateState) Task=`508a950e-4244-4278-8b95-c91299a3121b`::moving from state init -> state preparing >Thread-71518::INFO::2014-09-25 17:58:13,866::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71518::INFO::2014-09-25 17:58:13,867::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71518::DEBUG::2014-09-25 17:58:13,867::task::1191::Storage.TaskManager.Task::(prepare) Task=`508a950e-4244-4278-8b95-c91299a3121b`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71518::DEBUG::2014-09-25 17:58:13,867::task::595::Storage.TaskManager.Task::(_updateState) Task=`508a950e-4244-4278-8b95-c91299a3121b`::moving from state preparing -> state finished >Thread-71518::DEBUG::2014-09-25 17:58:13,867::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71518::DEBUG::2014-09-25 17:58:13,868::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71518::DEBUG::2014-09-25 17:58:13,868::task::993::Storage.TaskManager.Task::(_decref) Task=`508a950e-4244-4278-8b95-c91299a3121b`::ref 0 aborting False >Thread-71518::DEBUG::2014-09-25 17:58:13,868::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71518::DEBUG::2014-09-25 17:58:13,869::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:13,883::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:13,884::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71519::DEBUG::2014-09-25 17:58:13,884::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71519::DEBUG::2014-09-25 17:58:13,886::task::595::Storage.TaskManager.Task::(_updateState) Task=`5334191c-83ce-4e93-beff-eaaae46d04da`::moving from state init -> state preparing >Thread-71519::INFO::2014-09-25 17:58:13,886::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71519::DEBUG::2014-09-25 17:58:13,887::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`88a5fe07-abd4-4936-9efd-753686bd7b1f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71519::DEBUG::2014-09-25 17:58:13,887::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71519::DEBUG::2014-09-25 17:58:13,888::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71519::DEBUG::2014-09-25 17:58:13,888::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`88a5fe07-abd4-4936-9efd-753686bd7b1f`::Granted request >Thread-71519::DEBUG::2014-09-25 17:58:13,888::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5334191c-83ce-4e93-beff-eaaae46d04da`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71519::DEBUG::2014-09-25 17:58:13,889::task::993::Storage.TaskManager.Task::(_decref) Task=`5334191c-83ce-4e93-beff-eaaae46d04da`::ref 1 aborting False >Thread-71519::INFO::2014-09-25 17:58:13,893::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71519::DEBUG::2014-09-25 17:58:13,893::task::1191::Storage.TaskManager.Task::(prepare) Task=`5334191c-83ce-4e93-beff-eaaae46d04da`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71519::DEBUG::2014-09-25 17:58:13,893::task::595::Storage.TaskManager.Task::(_updateState) Task=`5334191c-83ce-4e93-beff-eaaae46d04da`::moving from state preparing -> state finished >Thread-71519::DEBUG::2014-09-25 17:58:13,894::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71519::DEBUG::2014-09-25 17:58:13,894::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71519::DEBUG::2014-09-25 17:58:13,894::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71519::DEBUG::2014-09-25 17:58:13,895::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71519::DEBUG::2014-09-25 17:58:13,895::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71519::DEBUG::2014-09-25 17:58:13,895::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71519::DEBUG::2014-09-25 17:58:13,895::task::993::Storage.TaskManager.Task::(_decref) Task=`5334191c-83ce-4e93-beff-eaaae46d04da`::ref 0 aborting False >Thread-71519::DEBUG::2014-09-25 17:58:13,896::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71519::DEBUG::2014-09-25 17:58:13,896::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:14,245::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:14,246::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71520::DEBUG::2014-09-25 17:58:14,248::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:15,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:15,433::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151071 s, 67.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:15,657::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:15,657::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:15,658::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:15,658::__init__::232::IOProcess::(_processLogs) (110038) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:15,660::__init__::232::IOProcess::(_processLogs) (110038) Queuing response >Thread-81::DEBUG::2014-09-25 17:58:15,661::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:58:15,674::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00302855 s, 123 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:15,675::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:15,676::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:15,676::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:15,676::__init__::232::IOProcess::(_processLogs) (110039) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:15,678::__init__::232::IOProcess::(_processLogs) (110039) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:17,260::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:17,262::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71521::DEBUG::2014-09-25 17:58:17,263::task::595::Storage.TaskManager.Task::(_updateState) Task=`53007f54-d21e-44b7-a346-bc825bd793e7`::moving from state init -> state preparing >Thread-71521::INFO::2014-09-25 17:58:17,263::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71521::INFO::2014-09-25 17:58:17,264::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000470116', 'lastCheck': '5.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000530981', 'lastCheck': '9.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00302855', 'lastCheck': '1.6', 'valid': True}} >Thread-71521::DEBUG::2014-09-25 17:58:17,264::task::1191::Storage.TaskManager.Task::(prepare) Task=`53007f54-d21e-44b7-a346-bc825bd793e7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000470116', 'lastCheck': '5.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000530981', 'lastCheck': '9.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00302855', 'lastCheck': '1.6', 'valid': True}} >Thread-71521::DEBUG::2014-09-25 17:58:17,264::task::595::Storage.TaskManager.Task::(_updateState) Task=`53007f54-d21e-44b7-a346-bc825bd793e7`::moving from state preparing -> state finished >Thread-71521::DEBUG::2014-09-25 17:58:17,264::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71521::DEBUG::2014-09-25 17:58:17,264::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71521::DEBUG::2014-09-25 17:58:17,264::task::993::Storage.TaskManager.Task::(_decref) Task=`53007f54-d21e-44b7-a346-bc825bd793e7`::ref 0 aborting False >Thread-71521::DEBUG::2014-09-25 17:58:17,266::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:17,286::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:17,288::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71522::DEBUG::2014-09-25 17:58:17,292::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:17,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:17,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0124216 s, 82.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:17,620::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:17,620::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:17,620::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:17,620::__init__::232::IOProcess::(_processLogs) (110040) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:17,621::__init__::232::IOProcess::(_processLogs) (110040) Queuing response >Thread-80::DEBUG::2014-09-25 17:58:17,622::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:58:17,631::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000481129 s, 786 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:17,632::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:17,632::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:17,633::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:17,633::__init__::232::IOProcess::(_processLogs) (110041) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:17,634::__init__::232::IOProcess::(_processLogs) (110041) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:19,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:19,507::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133894 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:20,350::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:20,352::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71523::DEBUG::2014-09-25 17:58:20,354::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:58:21,522::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:21,522::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:21,522::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:21,523::__init__::232::IOProcess::(_processLogs) (110042) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:21,523::__init__::232::IOProcess::(_processLogs) (110042) Queuing response >Thread-79::DEBUG::2014-09-25 17:58:21,524::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:21,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-79::DEBUG::2014-09-25 17:58:21,540::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000423366 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:21,541::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:21,542::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:21,542::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:21,542::__init__::232::IOProcess::(_processLogs) (110043) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:21,552::__init__::232::IOProcess::(_processLogs) (110043) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:21,555::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:21,556::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:21,558::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:21,559::__init__::232::IOProcess::(_processLogs) (110044) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:21,561::__init__::232::IOProcess::(_processLogs) (110044) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:21,564::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:21,566::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:21,567::__init__::232::IOProcess::(_processLogs) Extracting request information... >Dummy-148::DEBUG::2014-09-25 17:58:21,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0266377 s, 38.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:21,569::__init__::232::IOProcess::(_processLogs) (110045) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:21,570::__init__::232::IOProcess::(_processLogs) (110045) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:21,573::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:21,575::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:21,576::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:21,577::__init__::232::IOProcess::(_processLogs) (110046) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:21,578::__init__::232::IOProcess::(_processLogs) (110046) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:23,364::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:23,365::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71524::DEBUG::2014-09-25 17:58:23,367::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:23,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:23,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135578 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:23,927::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71525::DEBUG::2014-09-25 17:58:23,929::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:58:23,929::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71525::DEBUG::2014-09-25 17:58:23,931::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b910430-3acf-44ac-a716-75a6becf79c5`::moving from state init -> state preparing >Thread-71525::INFO::2014-09-25 17:58:23,932::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71525::INFO::2014-09-25 17:58:23,932::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71525::DEBUG::2014-09-25 17:58:23,933::task::1191::Storage.TaskManager.Task::(prepare) Task=`1b910430-3acf-44ac-a716-75a6becf79c5`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71525::DEBUG::2014-09-25 17:58:23,933::task::595::Storage.TaskManager.Task::(_updateState) Task=`1b910430-3acf-44ac-a716-75a6becf79c5`::moving from state preparing -> state finished >Thread-71525::DEBUG::2014-09-25 17:58:23,933::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71525::DEBUG::2014-09-25 17:58:23,933::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71525::DEBUG::2014-09-25 17:58:23,934::task::993::Storage.TaskManager.Task::(_decref) Task=`1b910430-3acf-44ac-a716-75a6becf79c5`::ref 0 aborting False >Thread-71525::DEBUG::2014-09-25 17:58:23,934::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71525::DEBUG::2014-09-25 17:58:23,934::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:23,948::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71526::DEBUG::2014-09-25 17:58:23,949::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:58:23,949::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71526::DEBUG::2014-09-25 17:58:23,951::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d577697-0be1-4869-9b15-d30c2ff09f07`::moving from state init -> state preparing >Thread-71526::INFO::2014-09-25 17:58:23,951::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71526::DEBUG::2014-09-25 17:58:23,952::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7d530af1-c65a-4f6c-b6fb-73e555878fe3`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71526::DEBUG::2014-09-25 17:58:23,952::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71526::DEBUG::2014-09-25 17:58:23,953::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71526::DEBUG::2014-09-25 17:58:23,953::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`7d530af1-c65a-4f6c-b6fb-73e555878fe3`::Granted request >Thread-71526::DEBUG::2014-09-25 17:58:23,954::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`8d577697-0be1-4869-9b15-d30c2ff09f07`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71526::DEBUG::2014-09-25 17:58:23,954::task::993::Storage.TaskManager.Task::(_decref) Task=`8d577697-0be1-4869-9b15-d30c2ff09f07`::ref 1 aborting False >Thread-71526::INFO::2014-09-25 17:58:23,959::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71526::DEBUG::2014-09-25 17:58:23,959::task::1191::Storage.TaskManager.Task::(prepare) Task=`8d577697-0be1-4869-9b15-d30c2ff09f07`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71526::DEBUG::2014-09-25 17:58:23,960::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d577697-0be1-4869-9b15-d30c2ff09f07`::moving from state preparing -> state finished >Thread-71526::DEBUG::2014-09-25 17:58:23,960::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71526::DEBUG::2014-09-25 17:58:23,960::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71526::DEBUG::2014-09-25 17:58:23,961::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71526::DEBUG::2014-09-25 17:58:23,961::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71526::DEBUG::2014-09-25 17:58:23,961::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71526::DEBUG::2014-09-25 17:58:23,962::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71526::DEBUG::2014-09-25 17:58:23,962::task::993::Storage.TaskManager.Task::(_decref) Task=`8d577697-0be1-4869-9b15-d30c2ff09f07`::ref 0 aborting False >Thread-71526::DEBUG::2014-09-25 17:58:23,963::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71526::DEBUG::2014-09-25 17:58:23,964::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:25,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:25,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132483 s, 77.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:25,680::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:25,681::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:25,681::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:25,681::__init__::232::IOProcess::(_processLogs) (110047) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:25,681::__init__::232::IOProcess::(_processLogs) (110047) Queuing response >Thread-81::DEBUG::2014-09-25 17:58:25,683::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:58:25,691::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000341241 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:25,693::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:25,693::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:25,693::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:25,693::__init__::232::IOProcess::(_processLogs) (110048) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:25,694::__init__::232::IOProcess::(_processLogs) (110048) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:26,384::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:26,385::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71527::DEBUG::2014-09-25 17:58:26,387::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:58:27,635::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:27,636::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:27,636::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:27,636::__init__::232::IOProcess::(_processLogs) (110049) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:27,637::__init__::232::IOProcess::(_processLogs) (110049) Queuing response >Thread-80::DEBUG::2014-09-25 17:58:27,638::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:58:27,647::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000495326 s, 763 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:27,648::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:27,648::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:27,649::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:27,649::__init__::232::IOProcess::(_processLogs) (110050) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:27,650::__init__::232::IOProcess::(_processLogs) (110050) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:27,681::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:27,702::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133946 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:29,401::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:29,403::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71528::DEBUG::2014-09-25 17:58:29,405::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:29,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:29,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013511 s, 75.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:31,575::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:31,576::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:31,576::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:31,576::__init__::232::IOProcess::(_processLogs) (110051) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:31,578::__init__::232::IOProcess::(_processLogs) (110051) Queuing response >Thread-79::DEBUG::2014-09-25 17:58:31,578::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:58:31,591::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00318596 s, 176 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:31,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:31,593::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:31,593::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:31,593::__init__::232::IOProcess::(_processLogs) (110052) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:31,596::__init__::232::IOProcess::(_processLogs) (110052) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:31,597::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:31,598::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:31,598::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:31,599::__init__::232::IOProcess::(_processLogs) (110053) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:31,599::__init__::232::IOProcess::(_processLogs) (110053) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:31,599::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:31,600::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:31,600::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:31,600::__init__::232::IOProcess::(_processLogs) (110054) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:31,601::__init__::232::IOProcess::(_processLogs) (110054) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:31,602::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:31,602::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:31,602::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:31,603::__init__::232::IOProcess::(_processLogs) (110055) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:31,604::__init__::232::IOProcess::(_processLogs) (110055) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:31,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:31,794::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168003 s, 61.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:32,416::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:32,417::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71529::DEBUG::2014-09-25 17:58:32,420::task::595::Storage.TaskManager.Task::(_updateState) Task=`07268259-6df6-4560-a1d7-be6d9618d07b`::moving from state init -> state preparing >Thread-71529::INFO::2014-09-25 17:58:32,420::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71529::INFO::2014-09-25 17:58:32,421::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00318596', 'lastCheck': '0.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000495326', 'lastCheck': '4.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000341241', 'lastCheck': '6.7', 'valid': True}} >Thread-71529::DEBUG::2014-09-25 17:58:32,421::task::1191::Storage.TaskManager.Task::(prepare) Task=`07268259-6df6-4560-a1d7-be6d9618d07b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00318596', 'lastCheck': '0.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000495326', 'lastCheck': '4.8', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000341241', 'lastCheck': '6.7', 'valid': True}} >Thread-71529::DEBUG::2014-09-25 17:58:32,421::task::595::Storage.TaskManager.Task::(_updateState) Task=`07268259-6df6-4560-a1d7-be6d9618d07b`::moving from state preparing -> state finished >Thread-71529::DEBUG::2014-09-25 17:58:32,422::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71529::DEBUG::2014-09-25 17:58:32,422::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71529::DEBUG::2014-09-25 17:58:32,422::task::993::Storage.TaskManager.Task::(_decref) Task=`07268259-6df6-4560-a1d7-be6d9618d07b`::ref 0 aborting False >Thread-71529::DEBUG::2014-09-25 17:58:32,424::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:32,443::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:32,444::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71530::DEBUG::2014-09-25 17:58:32,446::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:33,814::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:33,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145398 s, 70.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:33,994::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:33,996::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71531::DEBUG::2014-09-25 17:58:33,996::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71531::DEBUG::2014-09-25 17:58:33,998::task::595::Storage.TaskManager.Task::(_updateState) Task=`276c480d-1a4e-4e08-94cd-9abef67ada12`::moving from state init -> state preparing >Thread-71531::INFO::2014-09-25 17:58:33,998::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71531::INFO::2014-09-25 17:58:33,999::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71531::DEBUG::2014-09-25 17:58:33,999::task::1191::Storage.TaskManager.Task::(prepare) Task=`276c480d-1a4e-4e08-94cd-9abef67ada12`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71531::DEBUG::2014-09-25 17:58:33,999::task::595::Storage.TaskManager.Task::(_updateState) Task=`276c480d-1a4e-4e08-94cd-9abef67ada12`::moving from state preparing -> state finished >Thread-71531::DEBUG::2014-09-25 17:58:33,999::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71531::DEBUG::2014-09-25 17:58:34,000::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71531::DEBUG::2014-09-25 17:58:34,000::task::993::Storage.TaskManager.Task::(_decref) Task=`276c480d-1a4e-4e08-94cd-9abef67ada12`::ref 0 aborting False >Thread-71531::DEBUG::2014-09-25 17:58:34,000::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71531::DEBUG::2014-09-25 17:58:34,001::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:34,017::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:34,018::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71532::DEBUG::2014-09-25 17:58:34,018::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71532::DEBUG::2014-09-25 17:58:34,020::task::595::Storage.TaskManager.Task::(_updateState) Task=`ea8f433e-e7ef-481d-8f16-10248bad90f6`::moving from state init -> state preparing >Thread-71532::INFO::2014-09-25 17:58:34,021::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71532::DEBUG::2014-09-25 17:58:34,021::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`945fceca-e27c-462e-8515-44f460509f3e`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71532::DEBUG::2014-09-25 17:58:34,022::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71532::DEBUG::2014-09-25 17:58:34,022::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71532::DEBUG::2014-09-25 17:58:34,022::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`945fceca-e27c-462e-8515-44f460509f3e`::Granted request >Thread-71532::DEBUG::2014-09-25 17:58:34,023::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ea8f433e-e7ef-481d-8f16-10248bad90f6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71532::DEBUG::2014-09-25 17:58:34,023::task::993::Storage.TaskManager.Task::(_decref) Task=`ea8f433e-e7ef-481d-8f16-10248bad90f6`::ref 1 aborting False >Thread-71532::INFO::2014-09-25 17:58:34,027::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71532::DEBUG::2014-09-25 17:58:34,028::task::1191::Storage.TaskManager.Task::(prepare) Task=`ea8f433e-e7ef-481d-8f16-10248bad90f6`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71532::DEBUG::2014-09-25 17:58:34,028::task::595::Storage.TaskManager.Task::(_updateState) Task=`ea8f433e-e7ef-481d-8f16-10248bad90f6`::moving from state preparing -> state finished >Thread-71532::DEBUG::2014-09-25 17:58:34,028::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71532::DEBUG::2014-09-25 17:58:34,028::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71532::DEBUG::2014-09-25 17:58:34,029::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71532::DEBUG::2014-09-25 17:58:34,029::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71532::DEBUG::2014-09-25 17:58:34,029::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71532::DEBUG::2014-09-25 17:58:34,029::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71532::DEBUG::2014-09-25 17:58:34,030::task::993::Storage.TaskManager.Task::(_decref) Task=`ea8f433e-e7ef-481d-8f16-10248bad90f6`::ref 0 aborting False >Thread-71532::DEBUG::2014-09-25 17:58:34,030::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71532::DEBUG::2014-09-25 17:58:34,031::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:35,495::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:35,497::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71533::DEBUG::2014-09-25 17:58:35,498::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:58:35,696::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:35,696::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:35,696::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:35,697::__init__::232::IOProcess::(_processLogs) (110056) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:35,697::__init__::232::IOProcess::(_processLogs) (110056) Queuing response >Thread-81::DEBUG::2014-09-25 17:58:35,698::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:58:35,707::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000325 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:35,708::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:35,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:35,709::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:35,710::__init__::232::IOProcess::(_processLogs) (110057) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:35,711::__init__::232::IOProcess::(_processLogs) (110057) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:35,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:35,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132972 s, 77.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:37,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:37,652::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:37,652::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:37,652::__init__::232::IOProcess::(_processLogs) (110058) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:37,653::__init__::232::IOProcess::(_processLogs) (110058) Queuing response >Thread-80::DEBUG::2014-09-25 17:58:37,654::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:58:37,662::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000320647 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:37,664::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:37,664::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:37,665::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:37,666::__init__::232::IOProcess::(_processLogs) (110059) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:37,666::__init__::232::IOProcess::(_processLogs) (110059) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:37,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:37,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136448 s, 75.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:38,512::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:38,514::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71534::DEBUG::2014-09-25 17:58:38,516::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:39,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:39,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134585 s, 76.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:41,532::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:41,534::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71535::DEBUG::2014-09-25 17:58:41,536::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:58:41,604::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:41,604::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:41,605::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:41,605::__init__::232::IOProcess::(_processLogs) (110060) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:41,606::__init__::232::IOProcess::(_processLogs) (110060) Queuing response >Thread-79::DEBUG::2014-09-25 17:58:41,607::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:58:41,630::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00990002 s, 56.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:41,632::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:41,632::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:41,632::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:41,632::__init__::232::IOProcess::(_processLogs) (110061) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:41,636::__init__::232::IOProcess::(_processLogs) (110061) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:41,637::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:41,637::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:41,638::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:41,638::__init__::232::IOProcess::(_processLogs) (110062) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:41,639::__init__::232::IOProcess::(_processLogs) (110062) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:41,639::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:41,639::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:41,640::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:41,640::__init__::232::IOProcess::(_processLogs) (110063) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:41,640::__init__::232::IOProcess::(_processLogs) (110063) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:41,641::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:41,641::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:41,642::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:41,642::__init__::232::IOProcess::(_processLogs) (110064) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:41,642::__init__::232::IOProcess::(_processLogs) (110064) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:41,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:42,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135732 s, 75.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:58:44,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:44,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135596 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:44,064::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71536::DEBUG::2014-09-25 17:58:44,067::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:58:44,067::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71536::DEBUG::2014-09-25 17:58:44,072::task::595::Storage.TaskManager.Task::(_updateState) Task=`0610f3a3-0cb7-4214-a924-6f60acb74db3`::moving from state init -> state preparing >Thread-71536::INFO::2014-09-25 17:58:44,073::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71536::INFO::2014-09-25 17:58:44,075::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71536::DEBUG::2014-09-25 17:58:44,076::task::1191::Storage.TaskManager.Task::(prepare) Task=`0610f3a3-0cb7-4214-a924-6f60acb74db3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71536::DEBUG::2014-09-25 17:58:44,077::task::595::Storage.TaskManager.Task::(_updateState) Task=`0610f3a3-0cb7-4214-a924-6f60acb74db3`::moving from state preparing -> state finished >Thread-71536::DEBUG::2014-09-25 17:58:44,077::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71536::DEBUG::2014-09-25 17:58:44,078::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71536::DEBUG::2014-09-25 17:58:44,079::task::993::Storage.TaskManager.Task::(_decref) Task=`0610f3a3-0cb7-4214-a924-6f60acb74db3`::ref 0 aborting False >Thread-71536::DEBUG::2014-09-25 17:58:44,080::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71536::DEBUG::2014-09-25 17:58:44,082::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:44,097::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71537::DEBUG::2014-09-25 17:58:44,098::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:58:44,098::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71537::DEBUG::2014-09-25 17:58:44,100::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e6d544a-c62e-41d2-927c-e573fd379d3b`::moving from state init -> state preparing >Thread-71537::INFO::2014-09-25 17:58:44,100::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71537::DEBUG::2014-09-25 17:58:44,101::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b006669d-a2e7-4a44-92ed-512cd9ecce87`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71537::DEBUG::2014-09-25 17:58:44,101::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71537::DEBUG::2014-09-25 17:58:44,102::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71537::DEBUG::2014-09-25 17:58:44,102::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b006669d-a2e7-4a44-92ed-512cd9ecce87`::Granted request >Thread-71537::DEBUG::2014-09-25 17:58:44,103::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5e6d544a-c62e-41d2-927c-e573fd379d3b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71537::DEBUG::2014-09-25 17:58:44,103::task::993::Storage.TaskManager.Task::(_decref) Task=`5e6d544a-c62e-41d2-927c-e573fd379d3b`::ref 1 aborting False >Thread-71537::INFO::2014-09-25 17:58:44,107::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71537::DEBUG::2014-09-25 17:58:44,107::task::1191::Storage.TaskManager.Task::(prepare) Task=`5e6d544a-c62e-41d2-927c-e573fd379d3b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71537::DEBUG::2014-09-25 17:58:44,108::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e6d544a-c62e-41d2-927c-e573fd379d3b`::moving from state preparing -> state finished >Thread-71537::DEBUG::2014-09-25 17:58:44,108::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71537::DEBUG::2014-09-25 17:58:44,108::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71537::DEBUG::2014-09-25 17:58:44,109::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71537::DEBUG::2014-09-25 17:58:44,109::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71537::DEBUG::2014-09-25 17:58:44,109::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71537::DEBUG::2014-09-25 17:58:44,109::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71537::DEBUG::2014-09-25 17:58:44,110::task::993::Storage.TaskManager.Task::(_decref) Task=`5e6d544a-c62e-41d2-927c-e573fd379d3b`::ref 0 aborting False >Thread-71537::DEBUG::2014-09-25 17:58:44,110::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7239368704', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71537::DEBUG::2014-09-25 17:58:44,111::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:44,551::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:44,552::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71538::DEBUG::2014-09-25 17:58:44,554::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:58:45,711::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:45,712::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:45,712::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:45,712::__init__::232::IOProcess::(_processLogs) (110065) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:45,713::__init__::232::IOProcess::(_processLogs) (110065) Queuing response >Thread-81::DEBUG::2014-09-25 17:58:45,714::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:58:45,723::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000357852 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:45,724::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:45,724::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:45,725::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:45,725::__init__::232::IOProcess::(_processLogs) (110066) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:45,727::__init__::232::IOProcess::(_processLogs) (110066) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:46,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:46,124::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0221337 s, 46.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:47,570::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:47,571::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71539::DEBUG::2014-09-25 17:58:47,573::task::595::Storage.TaskManager.Task::(_updateState) Task=`4cbd31c6-27e2-4d8c-bede-5ff9316e893e`::moving from state init -> state preparing >Thread-71539::INFO::2014-09-25 17:58:47,574::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71539::INFO::2014-09-25 17:58:47,574::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00990002', 'lastCheck': '5.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000320647', 'lastCheck': '9.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000357852', 'lastCheck': '1.8', 'valid': True}} >Thread-71539::DEBUG::2014-09-25 17:58:47,575::task::1191::Storage.TaskManager.Task::(prepare) Task=`4cbd31c6-27e2-4d8c-bede-5ff9316e893e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00990002', 'lastCheck': '5.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000320647', 'lastCheck': '9.9', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000357852', 'lastCheck': '1.8', 'valid': True}} >Thread-71539::DEBUG::2014-09-25 17:58:47,575::task::595::Storage.TaskManager.Task::(_updateState) Task=`4cbd31c6-27e2-4d8c-bede-5ff9316e893e`::moving from state preparing -> state finished >Thread-71539::DEBUG::2014-09-25 17:58:47,575::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71539::DEBUG::2014-09-25 17:58:47,575::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71539::DEBUG::2014-09-25 17:58:47,575::task::993::Storage.TaskManager.Task::(_decref) Task=`4cbd31c6-27e2-4d8c-bede-5ff9316e893e`::ref 0 aborting False >Thread-71539::DEBUG::2014-09-25 17:58:47,578::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:47,598::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:47,599::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71540::DEBUG::2014-09-25 17:58:47,601::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:58:47,667::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:47,667::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:47,667::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:47,668::__init__::232::IOProcess::(_processLogs) (110067) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:47,668::__init__::232::IOProcess::(_processLogs) (110067) Queuing response >Thread-80::DEBUG::2014-09-25 17:58:47,669::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:58:47,678::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000423038 s, 894 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:47,679::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:47,679::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:47,680::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:47,680::__init__::232::IOProcess::(_processLogs) (110068) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:47,681::__init__::232::IOProcess::(_processLogs) (110068) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:48,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:48,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134573 s, 76.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:58:50,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:50,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134052 s, 76.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:50,655::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:50,657::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71541::DEBUG::2014-09-25 17:58:50,659::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-79::DEBUG::2014-09-25 17:58:51,643::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:58:51,646::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,648::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,648::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,648::__init__::232::IOProcess::(_processLogs) (110069) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:58:51,648::__init__::232::IOProcess::(_processLogs) (110069) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:51,650::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,650::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,650::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,650::__init__::232::IOProcess::(_processLogs) (110070) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:58:51,651::__init__::232::IOProcess::(_processLogs) (110070) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:51,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,652::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,652::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,652::__init__::232::IOProcess::(_processLogs) (110071) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:58:51,653::__init__::232::IOProcess::(_processLogs) (110071) Queuing response >Thread-79::DEBUG::2014-09-25 17:58:51,657::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 17:58:51,658::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,658::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,659::__init__::232::IOProcess::(_processLogs) (110072) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:58:51,682::__init__::232::IOProcess::(_processLogs) (110072) Queuing response >Thread-79::DEBUG::2014-09-25 17:58:51,683::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:58:51,684::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,684::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,685::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,685::__init__::232::IOProcess::(_processLogs) (110073) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:58:51,685::__init__::232::IOProcess::(_processLogs) (110073) Queuing response >Thread-79::DEBUG::2014-09-25 17:58:51,686::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 17:58:51,687::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,688::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,688::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,688::__init__::232::IOProcess::(_processLogs) (110074) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:51,688::__init__::232::IOProcess::(_processLogs) (110074) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:51,689::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,689::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,690::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,690::__init__::232::IOProcess::(_processLogs) (110075) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:58:51,690::__init__::232::IOProcess::(_processLogs) (110075) Queuing response >Thread-79::DEBUG::2014-09-25 17:58:51,691::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 17:58:51,691::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 17:58:51,692::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:58:51,692::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,693::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,693::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,693::__init__::232::IOProcess::(_processLogs) (110076) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:51,694::__init__::232::IOProcess::(_processLogs) (110076) Queuing response >Thread-79::DEBUG::2014-09-25 17:58:51,695::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:58:51,704::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000436564 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:51,705::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,705::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,706::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,706::__init__::232::IOProcess::(_processLogs) (110077) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:51,707::__init__::232::IOProcess::(_processLogs) (110077) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:51,708::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,709::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,709::__init__::232::IOProcess::(_processLogs) (110078) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:51,709::__init__::232::IOProcess::(_processLogs) (110078) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:51,710::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,710::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,710::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,711::__init__::232::IOProcess::(_processLogs) (110079) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:51,711::__init__::232::IOProcess::(_processLogs) (110079) Queuing response >Thread-29::DEBUG::2014-09-25 17:58:51,712::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:51,712::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:51,713::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:51,714::__init__::232::IOProcess::(_processLogs) (110080) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:58:51,714::__init__::232::IOProcess::(_processLogs) (110080) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:52,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:52,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131147 s, 78.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:53,673::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:53,674::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71545::DEBUG::2014-09-25 17:58:53,676::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:54,134::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:54,136::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71546::DEBUG::2014-09-25 17:58:54,136::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71546::DEBUG::2014-09-25 17:58:54,137::task::595::Storage.TaskManager.Task::(_updateState) Task=`4db15439-88ca-463a-aa65-759bf7f0e1eb`::moving from state init -> state preparing >Thread-71546::INFO::2014-09-25 17:58:54,138::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71546::INFO::2014-09-25 17:58:54,138::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71546::DEBUG::2014-09-25 17:58:54,138::task::1191::Storage.TaskManager.Task::(prepare) Task=`4db15439-88ca-463a-aa65-759bf7f0e1eb`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71546::DEBUG::2014-09-25 17:58:54,139::task::595::Storage.TaskManager.Task::(_updateState) Task=`4db15439-88ca-463a-aa65-759bf7f0e1eb`::moving from state preparing -> state finished >Thread-71546::DEBUG::2014-09-25 17:58:54,139::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71546::DEBUG::2014-09-25 17:58:54,139::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71546::DEBUG::2014-09-25 17:58:54,139::task::993::Storage.TaskManager.Task::(_decref) Task=`4db15439-88ca-463a-aa65-759bf7f0e1eb`::ref 0 aborting False >Thread-71546::DEBUG::2014-09-25 17:58:54,140::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71546::DEBUG::2014-09-25 17:58:54,140::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:54,150::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71547::DEBUG::2014-09-25 17:58:54,151::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:58:54,151::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71547::DEBUG::2014-09-25 17:58:54,153::task::595::Storage.TaskManager.Task::(_updateState) Task=`286014c8-515b-4468-83cb-d73ce7a868bb`::moving from state init -> state preparing >Thread-71547::INFO::2014-09-25 17:58:54,153::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71547::DEBUG::2014-09-25 17:58:54,154::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6c0e8696-29c0-43d7-a5d5-e6048ed68a44`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71547::DEBUG::2014-09-25 17:58:54,154::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71547::DEBUG::2014-09-25 17:58:54,155::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71547::DEBUG::2014-09-25 17:58:54,155::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6c0e8696-29c0-43d7-a5d5-e6048ed68a44`::Granted request >Thread-71547::DEBUG::2014-09-25 17:58:54,155::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`286014c8-515b-4468-83cb-d73ce7a868bb`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71547::DEBUG::2014-09-25 17:58:54,156::task::993::Storage.TaskManager.Task::(_decref) Task=`286014c8-515b-4468-83cb-d73ce7a868bb`::ref 1 aborting False >Thread-71547::INFO::2014-09-25 17:58:54,160::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71547::DEBUG::2014-09-25 17:58:54,160::task::1191::Storage.TaskManager.Task::(prepare) Task=`286014c8-515b-4468-83cb-d73ce7a868bb`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71547::DEBUG::2014-09-25 17:58:54,161::task::595::Storage.TaskManager.Task::(_updateState) Task=`286014c8-515b-4468-83cb-d73ce7a868bb`::moving from state preparing -> state finished >Thread-71547::DEBUG::2014-09-25 17:58:54,161::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71547::DEBUG::2014-09-25 17:58:54,161::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71547::DEBUG::2014-09-25 17:58:54,161::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71547::DEBUG::2014-09-25 17:58:54,162::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71547::DEBUG::2014-09-25 17:58:54,162::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71547::DEBUG::2014-09-25 17:58:54,162::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71547::DEBUG::2014-09-25 17:58:54,162::task::993::Storage.TaskManager.Task::(_decref) Task=`286014c8-515b-4468-83cb-d73ce7a868bb`::ref 0 aborting False >Thread-71547::DEBUG::2014-09-25 17:58:54,163::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71547::DEBUG::2014-09-25 17:58:54,163::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:58:54,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:54,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132733 s, 77.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:55,728::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:55,728::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:55,729::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 17:58:55,730::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:58:55,730::__init__::232::IOProcess::(_processLogs) (110081) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:55,739::__init__::232::IOProcess::(_processLogs) (110081) Queuing response >Thread-81::DEBUG::2014-09-25 17:58:55,740::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000345635 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:55,741::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:55,741::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:55,742::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:55,742::__init__::232::IOProcess::(_processLogs) (110082) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:55,742::__init__::232::IOProcess::(_processLogs) (110082) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:56,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:56,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129965 s, 78.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:56,716::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:56,717::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71548::DEBUG::2014-09-25 17:58:56,719::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:58:57,683::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:57,683::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:57,683::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:57,684::__init__::232::IOProcess::(_processLogs) (110083) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:57,684::__init__::232::IOProcess::(_processLogs) (110083) Queuing response >Thread-80::DEBUG::2014-09-25 17:58:57,685::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:58:57,694::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000379001 s, 997 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:58:57,695::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:58:57,696::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:58:57,696::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:58:57,696::__init__::232::IOProcess::(_processLogs) (110084) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:58:57,697::__init__::232::IOProcess::(_processLogs) (110084) Queuing response >Dummy-148::DEBUG::2014-09-25 17:58:58,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:58:58,528::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150849 s, 67.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:58:59,733::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:58:59,734::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71549::DEBUG::2014-09-25 17:58:59,736::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:00,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:00,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135085 s, 75.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:01,714::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:01,715::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:01,716::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 17:59:01,716::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:59:01,717::__init__::232::IOProcess::(_processLogs) (110085) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:01,726::__init__::232::IOProcess::(_processLogs) (110085) Queuing response >Thread-79::DEBUG::2014-09-25 17:59:01,728::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000357078 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:01,729::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:01,729::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:01,729::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:01,730::__init__::232::IOProcess::(_processLogs) (110086) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:01,730::__init__::232::IOProcess::(_processLogs) (110086) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:01,732::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:01,732::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:01,733::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:01,733::__init__::232::IOProcess::(_processLogs) (110087) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:01,733::__init__::232::IOProcess::(_processLogs) (110087) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:01,734::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:01,734::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:01,735::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:01,735::__init__::232::IOProcess::(_processLogs) (110088) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:01,736::__init__::232::IOProcess::(_processLogs) (110088) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:01,736::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:01,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:01,738::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:01,738::__init__::232::IOProcess::(_processLogs) (110089) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:01,738::__init__::232::IOProcess::(_processLogs) (110089) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:02,589::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:02,612::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134938 s, 75.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:02,750::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:02,751::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71550::DEBUG::2014-09-25 17:59:02,754::task::595::Storage.TaskManager.Task::(_updateState) Task=`f99dcef9-2853-453d-88fe-d791ccabe246`::moving from state init -> state preparing >Thread-71550::INFO::2014-09-25 17:59:02,755::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71550::INFO::2014-09-25 17:59:02,755::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000357078', 'lastCheck': '1.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000379001', 'lastCheck': '5.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000345635', 'lastCheck': '7.0', 'valid': True}} >Thread-71550::DEBUG::2014-09-25 17:59:02,755::task::1191::Storage.TaskManager.Task::(prepare) Task=`f99dcef9-2853-453d-88fe-d791ccabe246`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000357078', 'lastCheck': '1.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000379001', 'lastCheck': '5.1', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000345635', 'lastCheck': '7.0', 'valid': True}} >Thread-71550::DEBUG::2014-09-25 17:59:02,756::task::595::Storage.TaskManager.Task::(_updateState) Task=`f99dcef9-2853-453d-88fe-d791ccabe246`::moving from state preparing -> state finished >Thread-71550::DEBUG::2014-09-25 17:59:02,756::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71550::DEBUG::2014-09-25 17:59:02,756::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71550::DEBUG::2014-09-25 17:59:02,756::task::993::Storage.TaskManager.Task::(_decref) Task=`f99dcef9-2853-453d-88fe-d791ccabe246`::ref 0 aborting False >Thread-71550::DEBUG::2014-09-25 17:59:02,759::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:02,780::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:02,781::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71551::DEBUG::2014-09-25 17:59:02,783::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:04,186::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:04,188::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71552::DEBUG::2014-09-25 17:59:04,188::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71552::DEBUG::2014-09-25 17:59:04,190::task::595::Storage.TaskManager.Task::(_updateState) Task=`1130a2b7-1969-4adb-a03f-6bab8efa1307`::moving from state init -> state preparing >Thread-71552::INFO::2014-09-25 17:59:04,190::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71552::INFO::2014-09-25 17:59:04,191::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71552::DEBUG::2014-09-25 17:59:04,191::task::1191::Storage.TaskManager.Task::(prepare) Task=`1130a2b7-1969-4adb-a03f-6bab8efa1307`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71552::DEBUG::2014-09-25 17:59:04,191::task::595::Storage.TaskManager.Task::(_updateState) Task=`1130a2b7-1969-4adb-a03f-6bab8efa1307`::moving from state preparing -> state finished >Thread-71552::DEBUG::2014-09-25 17:59:04,192::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71552::DEBUG::2014-09-25 17:59:04,192::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71552::DEBUG::2014-09-25 17:59:04,192::task::993::Storage.TaskManager.Task::(_decref) Task=`1130a2b7-1969-4adb-a03f-6bab8efa1307`::ref 0 aborting False >Thread-71552::DEBUG::2014-09-25 17:59:04,192::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71552::DEBUG::2014-09-25 17:59:04,193::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:04,202::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71553::DEBUG::2014-09-25 17:59:04,203::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:59:04,203::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71553::DEBUG::2014-09-25 17:59:04,205::task::595::Storage.TaskManager.Task::(_updateState) Task=`17f13821-6662-4e1c-802b-b8ac6549d15b`::moving from state init -> state preparing >Thread-71553::INFO::2014-09-25 17:59:04,206::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71553::DEBUG::2014-09-25 17:59:04,206::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ea08eae1-63c1-4b86-a548-d9cc83433b55`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71553::DEBUG::2014-09-25 17:59:04,206::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71553::DEBUG::2014-09-25 17:59:04,207::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71553::DEBUG::2014-09-25 17:59:04,207::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`ea08eae1-63c1-4b86-a548-d9cc83433b55`::Granted request >Thread-71553::DEBUG::2014-09-25 17:59:04,208::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`17f13821-6662-4e1c-802b-b8ac6549d15b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71553::DEBUG::2014-09-25 17:59:04,208::task::993::Storage.TaskManager.Task::(_decref) Task=`17f13821-6662-4e1c-802b-b8ac6549d15b`::ref 1 aborting False >Thread-71553::INFO::2014-09-25 17:59:04,212::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71553::DEBUG::2014-09-25 17:59:04,213::task::1191::Storage.TaskManager.Task::(prepare) Task=`17f13821-6662-4e1c-802b-b8ac6549d15b`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71553::DEBUG::2014-09-25 17:59:04,213::task::595::Storage.TaskManager.Task::(_updateState) Task=`17f13821-6662-4e1c-802b-b8ac6549d15b`::moving from state preparing -> state finished >Thread-71553::DEBUG::2014-09-25 17:59:04,213::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71553::DEBUG::2014-09-25 17:59:04,213::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71553::DEBUG::2014-09-25 17:59:04,214::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71553::DEBUG::2014-09-25 17:59:04,214::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71553::DEBUG::2014-09-25 17:59:04,214::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71553::DEBUG::2014-09-25 17:59:04,214::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71553::DEBUG::2014-09-25 17:59:04,215::task::993::Storage.TaskManager.Task::(_decref) Task=`17f13821-6662-4e1c-802b-b8ac6549d15b`::ref 0 aborting False >Thread-71553::DEBUG::2014-09-25 17:59:04,215::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71553::DEBUG::2014-09-25 17:59:04,216::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:04,631::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:04,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135467 s, 75.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:05,745::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:05,745::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:05,745::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:05,746::__init__::232::IOProcess::(_processLogs) (110090) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:05,746::__init__::232::IOProcess::(_processLogs) (110090) Queuing response >Thread-81::DEBUG::2014-09-25 17:59:05,747::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:59:05,756::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000358389 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:05,758::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:05,758::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:05,758::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:05,758::__init__::232::IOProcess::(_processLogs) (110091) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:05,759::__init__::232::IOProcess::(_processLogs) (110091) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:05,845::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:05,846::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71554::DEBUG::2014-09-25 17:59:05,848::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:06,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:06,696::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133516 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:07,699::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:07,699::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:07,700::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:07,700::__init__::232::IOProcess::(_processLogs) (110092) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:07,700::__init__::232::IOProcess::(_processLogs) (110092) Queuing response >Thread-80::DEBUG::2014-09-25 17:59:07,701::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:59:07,710::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000466204 s, 811 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:07,711::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:07,712::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:07,712::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:07,712::__init__::232::IOProcess::(_processLogs) (110093) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:07,713::__init__::232::IOProcess::(_processLogs) (110093) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:08,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:08,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135625 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:08,863::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:08,865::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71555::DEBUG::2014-09-25 17:59:08,867::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:10,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:10,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164848 s, 62.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:11,738::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:11,739::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:11,739::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:11,739::__init__::232::IOProcess::(_processLogs) (110094) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:11,740::__init__::232::IOProcess::(_processLogs) (110094) Queuing response >Thread-79::DEBUG::2014-09-25 17:59:11,741::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:59:11,750::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000437765 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:11,751::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:11,752::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:11,752::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:11,752::__init__::232::IOProcess::(_processLogs) (110095) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:11,753::__init__::232::IOProcess::(_processLogs) (110095) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:11,754::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:11,755::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:11,755::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:11,756::__init__::232::IOProcess::(_processLogs) (110096) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:11,756::__init__::232::IOProcess::(_processLogs) (110096) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:11,756::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:11,757::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:11,757::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:11,758::__init__::232::IOProcess::(_processLogs) (110097) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:11,758::__init__::232::IOProcess::(_processLogs) (110097) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:11,758::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:11,759::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:11,760::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:11,760::__init__::232::IOProcess::(_processLogs) (110098) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:11,760::__init__::232::IOProcess::(_processLogs) (110098) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:11,884::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:11,886::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71556::DEBUG::2014-09-25 17:59:11,888::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:12,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:12,830::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135644 s, 75.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:14,237::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71557::DEBUG::2014-09-25 17:59:14,239::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 17:59:14,239::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71557::DEBUG::2014-09-25 17:59:14,242::task::595::Storage.TaskManager.Task::(_updateState) Task=`cda4e005-29aa-4608-8257-75280e6a35e7`::moving from state init -> state preparing >Thread-71557::INFO::2014-09-25 17:59:14,242::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71557::INFO::2014-09-25 17:59:14,243::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71557::DEBUG::2014-09-25 17:59:14,243::task::1191::Storage.TaskManager.Task::(prepare) Task=`cda4e005-29aa-4608-8257-75280e6a35e7`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71557::DEBUG::2014-09-25 17:59:14,243::task::595::Storage.TaskManager.Task::(_updateState) Task=`cda4e005-29aa-4608-8257-75280e6a35e7`::moving from state preparing -> state finished >Thread-71557::DEBUG::2014-09-25 17:59:14,243::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71557::DEBUG::2014-09-25 17:59:14,244::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71557::DEBUG::2014-09-25 17:59:14,244::task::993::Storage.TaskManager.Task::(_decref) Task=`cda4e005-29aa-4608-8257-75280e6a35e7`::ref 0 aborting False >Thread-71557::DEBUG::2014-09-25 17:59:14,244::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71557::DEBUG::2014-09-25 17:59:14,245::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:14,255::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:14,256::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71558::DEBUG::2014-09-25 17:59:14,256::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71558::DEBUG::2014-09-25 17:59:14,257::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f3c2c2d-1b41-40f0-8e00-e4f428f32c9f`::moving from state init -> state preparing >Thread-71558::INFO::2014-09-25 17:59:14,258::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71558::DEBUG::2014-09-25 17:59:14,259::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e6bca7f6-998f-4847-b4fd-0235d1777996`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71558::DEBUG::2014-09-25 17:59:14,259::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71558::DEBUG::2014-09-25 17:59:14,259::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71558::DEBUG::2014-09-25 17:59:14,259::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`e6bca7f6-998f-4847-b4fd-0235d1777996`::Granted request >Thread-71558::DEBUG::2014-09-25 17:59:14,260::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`8f3c2c2d-1b41-40f0-8e00-e4f428f32c9f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71558::DEBUG::2014-09-25 17:59:14,260::task::993::Storage.TaskManager.Task::(_decref) Task=`8f3c2c2d-1b41-40f0-8e00-e4f428f32c9f`::ref 1 aborting False >Thread-71558::INFO::2014-09-25 17:59:14,265::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71558::DEBUG::2014-09-25 17:59:14,265::task::1191::Storage.TaskManager.Task::(prepare) Task=`8f3c2c2d-1b41-40f0-8e00-e4f428f32c9f`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71558::DEBUG::2014-09-25 17:59:14,265::task::595::Storage.TaskManager.Task::(_updateState) Task=`8f3c2c2d-1b41-40f0-8e00-e4f428f32c9f`::moving from state preparing -> state finished >Thread-71558::DEBUG::2014-09-25 17:59:14,266::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71558::DEBUG::2014-09-25 17:59:14,266::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71558::DEBUG::2014-09-25 17:59:14,266::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71558::DEBUG::2014-09-25 17:59:14,267::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71558::DEBUG::2014-09-25 17:59:14,267::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71558::DEBUG::2014-09-25 17:59:14,267::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71558::DEBUG::2014-09-25 17:59:14,267::task::993::Storage.TaskManager.Task::(_decref) Task=`8f3c2c2d-1b41-40f0-8e00-e4f428f32c9f`::ref 0 aborting False >Thread-71558::DEBUG::2014-09-25 17:59:14,268::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71558::DEBUG::2014-09-25 17:59:14,268::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:14,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:14,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132912 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:14,907::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:14,909::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71559::DEBUG::2014-09-25 17:59:14,911::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 17:59:15,761::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:15,762::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:15,762::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:15,762::__init__::232::IOProcess::(_processLogs) (110099) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:15,763::__init__::232::IOProcess::(_processLogs) (110099) Queuing response >Thread-81::DEBUG::2014-09-25 17:59:15,764::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:59:15,773::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000412151 s, 900 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:15,774::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:15,774::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:15,775::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:15,775::__init__::232::IOProcess::(_processLogs) (110100) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:15,776::__init__::232::IOProcess::(_processLogs) (110100) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:16,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:16,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182809 s, 56.0 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 17:59:17,714::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:59:17,718::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:17,719::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:17,720::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:17,720::__init__::232::IOProcess::(_processLogs) (110101) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:59:17,720::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:17,721::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:17,721::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:17,721::__init__::232::IOProcess::(_processLogs) (110102) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:59:17,722::__init__::232::IOProcess::(_processLogs) (110102) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:17,724::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:17,725::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:17,727::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:17,729::__init__::232::IOProcess::(_processLogs) (110103) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 17:59:17,731::__init__::232::IOProcess::(_processLogs) (110101) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:17,733::__init__::232::IOProcess::(_processLogs) (110103) Queuing response >Thread-80::DEBUG::2014-09-25 17:59:17,736::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 17:59:17,737::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:17,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:17,737::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:17,737::__init__::232::IOProcess::(_processLogs) (110104) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 17:59:17,741::__init__::232::IOProcess::(_processLogs) (110104) Queuing response >Thread-80::DEBUG::2014-09-25 17:59:17,742::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 17:59:17,742::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:17,743::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:17,743::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:17,743::__init__::232::IOProcess::(_processLogs) (110105) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 17:59:17,744::__init__::232::IOProcess::(_processLogs) (110105) Queuing response >Thread-80::DEBUG::2014-09-25 17:59:17,744::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Iso', 'DESCRIPTION=ISO_DOMAIN', 'IOOPTIMEOUTSEC=1', 'LEASERETRIES=3', 'LEASETIMESEC=5', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', 'ROLE=Regular', 'SDUUID=fc587b6a-bac3-4668-99de-7f6fd6acce7f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=35c067da9cf5708013ca1455067504be59cf3476'] >Thread-29::DEBUG::2014-09-25 17:59:17,746::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:17,746::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:17,746::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:17,746::__init__::232::IOProcess::(_processLogs) (110106) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:17,747::__init__::232::IOProcess::(_processLogs) (110106) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:17,747::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:17,748::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:17,748::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:17,748::__init__::232::IOProcess::(_processLogs) (110107) Got request for method 'glob' >Thread-80::DEBUG::2014-09-25 17:59:17,749::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 17:59:17,749::__init__::232::IOProcess::(_processLogs) (110107) Queuing response >Thread-80::INFO::2014-09-25 17:59:17,750::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_imageNS already registered >Thread-80::INFO::2014-09-25 17:59:17,750::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace fc587b6a-bac3-4668-99de-7f6fd6acce7f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 17:59:17,751::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:17,751::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:17,752::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:17,752::__init__::232::IOProcess::(_processLogs) (110108) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:17,753::__init__::232::IOProcess::(_processLogs) (110108) Queuing response >Thread-80::DEBUG::2014-09-25 17:59:17,753::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:59:17,762::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000504377 s, 749 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:17,763::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:17,764::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:17,764::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:17,764::__init__::232::IOProcess::(_processLogs) (110109) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:17,765::__init__::232::IOProcess::(_processLogs) (110109) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:17,924::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:17,925::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71563::DEBUG::2014-09-25 17:59:17,928::task::595::Storage.TaskManager.Task::(_updateState) Task=`d304c842-29de-407a-a8b7-b309377e362d`::moving from state init -> state preparing >Thread-71563::INFO::2014-09-25 17:59:17,928::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71563::INFO::2014-09-25 17:59:17,929::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000437765', 'lastCheck': '6.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000504377', 'lastCheck': '0.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000412151', 'lastCheck': '2.2', 'valid': True}} >Thread-71563::DEBUG::2014-09-25 17:59:17,929::task::1191::Storage.TaskManager.Task::(prepare) Task=`d304c842-29de-407a-a8b7-b309377e362d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000437765', 'lastCheck': '6.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000504377', 'lastCheck': '0.2', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000412151', 'lastCheck': '2.2', 'valid': True}} >Thread-71563::DEBUG::2014-09-25 17:59:17,929::task::595::Storage.TaskManager.Task::(_updateState) Task=`d304c842-29de-407a-a8b7-b309377e362d`::moving from state preparing -> state finished >Thread-71563::DEBUG::2014-09-25 17:59:17,929::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71563::DEBUG::2014-09-25 17:59:17,930::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71563::DEBUG::2014-09-25 17:59:17,930::task::993::Storage.TaskManager.Task::(_decref) Task=`d304c842-29de-407a-a8b7-b309377e362d`::ref 0 aborting False >Thread-71563::DEBUG::2014-09-25 17:59:17,932::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:17,946::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:17,948::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71564::DEBUG::2014-09-25 17:59:17,949::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:19,017::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:19,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150531 s, 68.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:20,993::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:20,994::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71565::DEBUG::2014-09-25 17:59:20,996::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:21,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:21,084::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135051 s, 75.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:21,761::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:21,761::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:21,761::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:21,762::__init__::232::IOProcess::(_processLogs) (110110) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:21,762::__init__::232::IOProcess::(_processLogs) (110110) Queuing response >Thread-79::DEBUG::2014-09-25 17:59:21,763::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:59:21,772::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000434331 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:21,773::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:21,774::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:21,774::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:21,774::__init__::232::IOProcess::(_processLogs) (110111) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:21,775::__init__::232::IOProcess::(_processLogs) (110111) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:21,776::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:21,777::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:21,777::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:21,777::__init__::232::IOProcess::(_processLogs) (110112) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:21,777::__init__::232::IOProcess::(_processLogs) (110112) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:21,778::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:21,779::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:21,779::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:21,779::__init__::232::IOProcess::(_processLogs) (110113) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:21,779::__init__::232::IOProcess::(_processLogs) (110113) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:21,780::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:21,781::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:21,781::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:21,781::__init__::232::IOProcess::(_processLogs) (110114) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:21,781::__init__::232::IOProcess::(_processLogs) (110114) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:23,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:23,125::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132154 s, 77.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:24,009::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:24,011::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71566::DEBUG::2014-09-25 17:59:24,013::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:24,293::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:24,296::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71567::DEBUG::2014-09-25 17:59:24,296::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71567::DEBUG::2014-09-25 17:59:24,298::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a8c5066-f962-44af-84d9-32bbdb2e1b29`::moving from state init -> state preparing >Thread-71567::INFO::2014-09-25 17:59:24,299::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71567::INFO::2014-09-25 17:59:24,299::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71567::DEBUG::2014-09-25 17:59:24,299::task::1191::Storage.TaskManager.Task::(prepare) Task=`9a8c5066-f962-44af-84d9-32bbdb2e1b29`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71567::DEBUG::2014-09-25 17:59:24,300::task::595::Storage.TaskManager.Task::(_updateState) Task=`9a8c5066-f962-44af-84d9-32bbdb2e1b29`::moving from state preparing -> state finished >Thread-71567::DEBUG::2014-09-25 17:59:24,300::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71567::DEBUG::2014-09-25 17:59:24,300::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71567::DEBUG::2014-09-25 17:59:24,300::task::993::Storage.TaskManager.Task::(_decref) Task=`9a8c5066-f962-44af-84d9-32bbdb2e1b29`::ref 0 aborting False >Thread-71567::DEBUG::2014-09-25 17:59:24,301::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71567::DEBUG::2014-09-25 17:59:24,301::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:24,314::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:24,315::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71568::DEBUG::2014-09-25 17:59:24,315::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71568::DEBUG::2014-09-25 17:59:24,317::task::595::Storage.TaskManager.Task::(_updateState) Task=`b39d8630-6a17-41ee-8952-2526b0cd9748`::moving from state init -> state preparing >Thread-71568::INFO::2014-09-25 17:59:24,317::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71568::DEBUG::2014-09-25 17:59:24,318::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`295c49db-983c-47e6-877f-f77c0713c65c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71568::DEBUG::2014-09-25 17:59:24,318::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71568::DEBUG::2014-09-25 17:59:24,318::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71568::DEBUG::2014-09-25 17:59:24,319::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`295c49db-983c-47e6-877f-f77c0713c65c`::Granted request >Thread-71568::DEBUG::2014-09-25 17:59:24,319::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`b39d8630-6a17-41ee-8952-2526b0cd9748`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71568::DEBUG::2014-09-25 17:59:24,319::task::993::Storage.TaskManager.Task::(_decref) Task=`b39d8630-6a17-41ee-8952-2526b0cd9748`::ref 1 aborting False >Thread-71568::INFO::2014-09-25 17:59:24,324::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71568::DEBUG::2014-09-25 17:59:24,324::task::1191::Storage.TaskManager.Task::(prepare) Task=`b39d8630-6a17-41ee-8952-2526b0cd9748`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71568::DEBUG::2014-09-25 17:59:24,324::task::595::Storage.TaskManager.Task::(_updateState) Task=`b39d8630-6a17-41ee-8952-2526b0cd9748`::moving from state preparing -> state finished >Thread-71568::DEBUG::2014-09-25 17:59:24,324::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71568::DEBUG::2014-09-25 17:59:24,325::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71568::DEBUG::2014-09-25 17:59:24,325::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71568::DEBUG::2014-09-25 17:59:24,325::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71568::DEBUG::2014-09-25 17:59:24,326::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71568::DEBUG::2014-09-25 17:59:24,326::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71568::DEBUG::2014-09-25 17:59:24,326::task::993::Storage.TaskManager.Task::(_decref) Task=`b39d8630-6a17-41ee-8952-2526b0cd9748`::ref 0 aborting False >Thread-71568::DEBUG::2014-09-25 17:59:24,327::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7242514432', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71568::DEBUG::2014-09-25 17:59:24,327::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:25,145::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:25,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180995 s, 56.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:25,778::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:25,778::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:25,779::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:25,779::__init__::232::IOProcess::(_processLogs) (110115) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:25,783::__init__::232::IOProcess::(_processLogs) (110115) Queuing response >Thread-81::DEBUG::2014-09-25 17:59:25,784::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:59:25,800::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00313194 s, 118 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:25,802::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:25,802::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:25,802::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:25,802::__init__::232::IOProcess::(_processLogs) (110116) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:25,807::__init__::232::IOProcess::(_processLogs) (110116) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:27,027::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:27,028::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71569::DEBUG::2014-09-25 17:59:27,030::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:27,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:27,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151351 s, 67.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:27,767::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:27,767::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:27,768::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:27,768::__init__::232::IOProcess::(_processLogs) (110117) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:27,768::__init__::232::IOProcess::(_processLogs) (110117) Queuing response >Thread-80::DEBUG::2014-09-25 17:59:27,769::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:59:27,778::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000467063 s, 809 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:27,779::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:27,779::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:27,780::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:27,781::__init__::232::IOProcess::(_processLogs) (110118) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:27,782::__init__::232::IOProcess::(_processLogs) (110118) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:29,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:29,264::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130331 s, 78.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:30,046::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:30,048::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71570::DEBUG::2014-09-25 17:59:30,050::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:31,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:31,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194636 s, 52.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:31,783::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:31,783::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:31,783::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:31,784::__init__::232::IOProcess::(_processLogs) (110119) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:31,787::__init__::232::IOProcess::(_processLogs) (110119) Queuing response >Thread-79::DEBUG::2014-09-25 17:59:31,788::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:59:31,799::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00223642 s, 251 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:31,800::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:31,800::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:31,800::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:31,801::__init__::232::IOProcess::(_processLogs) (110120) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:31,803::__init__::232::IOProcess::(_processLogs) (110120) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:31,804::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:31,804::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:31,805::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:31,805::__init__::232::IOProcess::(_processLogs) (110121) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:31,805::__init__::232::IOProcess::(_processLogs) (110121) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:31,806::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:31,806::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:31,807::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:31,807::__init__::232::IOProcess::(_processLogs) (110122) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:31,807::__init__::232::IOProcess::(_processLogs) (110122) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:31,808::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:31,808::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:31,809::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:31,809::__init__::232::IOProcess::(_processLogs) (110123) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:31,809::__init__::232::IOProcess::(_processLogs) (110123) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:33,063::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:33,065::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71571::DEBUG::2014-09-25 17:59:33,067::task::595::Storage.TaskManager.Task::(_updateState) Task=`524a6e78-8dc8-496e-ae87-4f233bf7708f`::moving from state init -> state preparing >Thread-71571::INFO::2014-09-25 17:59:33,068::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71571::INFO::2014-09-25 17:59:33,068::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00223642', 'lastCheck': '1.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000467063', 'lastCheck': '5.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00313194', 'lastCheck': '7.3', 'valid': True}} >Thread-71571::DEBUG::2014-09-25 17:59:33,068::task::1191::Storage.TaskManager.Task::(prepare) Task=`524a6e78-8dc8-496e-ae87-4f233bf7708f`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00223642', 'lastCheck': '1.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000467063', 'lastCheck': '5.3', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00313194', 'lastCheck': '7.3', 'valid': True}} >Thread-71571::DEBUG::2014-09-25 17:59:33,069::task::595::Storage.TaskManager.Task::(_updateState) Task=`524a6e78-8dc8-496e-ae87-4f233bf7708f`::moving from state preparing -> state finished >Thread-71571::DEBUG::2014-09-25 17:59:33,069::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71571::DEBUG::2014-09-25 17:59:33,069::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71571::DEBUG::2014-09-25 17:59:33,069::task::993::Storage.TaskManager.Task::(_decref) Task=`524a6e78-8dc8-496e-ae87-4f233bf7708f`::ref 0 aborting False >Thread-71571::DEBUG::2014-09-25 17:59:33,072::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:33,086::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:33,088::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71572::DEBUG::2014-09-25 17:59:33,090::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:33,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:33,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186378 s, 54.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:34,353::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:34,355::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71573::DEBUG::2014-09-25 17:59:34,355::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71573::DEBUG::2014-09-25 17:59:34,357::task::595::Storage.TaskManager.Task::(_updateState) Task=`43775c87-c8a1-46d5-8e0f-e3617df4e880`::moving from state init -> state preparing >Thread-71573::INFO::2014-09-25 17:59:34,357::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71573::INFO::2014-09-25 17:59:34,358::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71573::DEBUG::2014-09-25 17:59:34,358::task::1191::Storage.TaskManager.Task::(prepare) Task=`43775c87-c8a1-46d5-8e0f-e3617df4e880`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71573::DEBUG::2014-09-25 17:59:34,358::task::595::Storage.TaskManager.Task::(_updateState) Task=`43775c87-c8a1-46d5-8e0f-e3617df4e880`::moving from state preparing -> state finished >Thread-71573::DEBUG::2014-09-25 17:59:34,358::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71573::DEBUG::2014-09-25 17:59:34,359::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71573::DEBUG::2014-09-25 17:59:34,359::task::993::Storage.TaskManager.Task::(_decref) Task=`43775c87-c8a1-46d5-8e0f-e3617df4e880`::ref 0 aborting False >Thread-71573::DEBUG::2014-09-25 17:59:34,359::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71573::DEBUG::2014-09-25 17:59:34,360::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 17:59:34,371::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 17:59:34,372::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71574::DEBUG::2014-09-25 17:59:34,373::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71574::DEBUG::2014-09-25 17:59:34,374::task::595::Storage.TaskManager.Task::(_updateState) Task=`4c2e5e79-7539-4a68-b3da-99d4af7f53bd`::moving from state init -> state preparing >Thread-71574::INFO::2014-09-25 17:59:34,374::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71574::DEBUG::2014-09-25 17:59:34,375::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`707963b2-0fbb-4d70-97ed-5857bcf341bd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71574::DEBUG::2014-09-25 17:59:34,375::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71574::DEBUG::2014-09-25 17:59:34,376::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71574::DEBUG::2014-09-25 17:59:34,376::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`707963b2-0fbb-4d70-97ed-5857bcf341bd`::Granted request >Thread-71574::DEBUG::2014-09-25 17:59:34,377::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4c2e5e79-7539-4a68-b3da-99d4af7f53bd`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71574::DEBUG::2014-09-25 17:59:34,377::task::993::Storage.TaskManager.Task::(_decref) Task=`4c2e5e79-7539-4a68-b3da-99d4af7f53bd`::ref 1 aborting False >Thread-71574::INFO::2014-09-25 17:59:34,381::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7241465856', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71574::DEBUG::2014-09-25 17:59:34,381::task::1191::Storage.TaskManager.Task::(prepare) Task=`4c2e5e79-7539-4a68-b3da-99d4af7f53bd`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7241465856', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71574::DEBUG::2014-09-25 17:59:34,382::task::595::Storage.TaskManager.Task::(_updateState) Task=`4c2e5e79-7539-4a68-b3da-99d4af7f53bd`::moving from state preparing -> state finished >Thread-71574::DEBUG::2014-09-25 17:59:34,382::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71574::DEBUG::2014-09-25 17:59:34,382::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71574::DEBUG::2014-09-25 17:59:34,383::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71574::DEBUG::2014-09-25 17:59:34,383::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71574::DEBUG::2014-09-25 17:59:34,383::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71574::DEBUG::2014-09-25 17:59:34,383::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71574::DEBUG::2014-09-25 17:59:34,384::task::993::Storage.TaskManager.Task::(_decref) Task=`4c2e5e79-7539-4a68-b3da-99d4af7f53bd`::ref 0 aborting False >Thread-71574::DEBUG::2014-09-25 17:59:34,384::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '7241465856', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75197579264', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71574::DEBUG::2014-09-25 17:59:34,385::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 17:59:35,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:35,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132389 s, 77.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:35,809::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:35,809::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:35,810::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:35,810::__init__::232::IOProcess::(_processLogs) (110124) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:35,812::__init__::232::IOProcess::(_processLogs) (110124) Queuing response >Thread-81::DEBUG::2014-09-25 17:59:35,813::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:59:35,826::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00147463 s, 252 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:35,827::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:35,828::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:35,828::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:35,828::__init__::232::IOProcess::(_processLogs) (110125) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:35,829::__init__::232::IOProcess::(_processLogs) (110125) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:37,432::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:37,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0197999 s, 51.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:37,782::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:37,782::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:37,783::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:37,783::__init__::232::IOProcess::(_processLogs) (110126) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:37,783::__init__::232::IOProcess::(_processLogs) (110126) Queuing response >Thread-80::DEBUG::2014-09-25 17:59:37,785::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 17:59:37,793::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000443321 s, 853 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:37,794::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:37,794::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:37,795::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:37,795::__init__::232::IOProcess::(_processLogs) (110127) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:37,795::__init__::232::IOProcess::(_processLogs) (110127) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:39,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:39,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132702 s, 77.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:59:41,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:41,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0267449 s, 38.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:41,810::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:41,810::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:41,810::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:41,811::__init__::232::IOProcess::(_processLogs) (110128) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:41,814::__init__::232::IOProcess::(_processLogs) (110128) Queuing response >Thread-79::DEBUG::2014-09-25 17:59:41,815::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:59:41,826::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00164639 s, 341 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:41,827::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:41,827::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:41,828::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:41,828::__init__::232::IOProcess::(_processLogs) (110129) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:41,829::__init__::232::IOProcess::(_processLogs) (110129) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:41,830::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:41,830::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:41,831::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:41,832::__init__::232::IOProcess::(_processLogs) (110130) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:41,832::__init__::232::IOProcess::(_processLogs) (110130) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:41,832::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:41,833::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:41,833::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:41,834::__init__::232::IOProcess::(_processLogs) (110131) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:41,834::__init__::232::IOProcess::(_processLogs) (110131) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:41,834::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:41,835::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:41,836::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:41,836::__init__::232::IOProcess::(_processLogs) (110132) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:41,836::__init__::232::IOProcess::(_processLogs) (110132) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:43,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:43,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188646 s, 54.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:59:45,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:45,672::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0254195 s, 40.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:45,831::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:45,832::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:45,832::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:45,832::__init__::232::IOProcess::(_processLogs) (110133) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:45,838::__init__::232::IOProcess::(_processLogs) (110133) Queuing response >Thread-81::DEBUG::2014-09-25 17:59:45,839::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 17:59:45,858::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00216754 s, 171 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:45,859::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:45,860::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:45,860::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:45,860::__init__::232::IOProcess::(_processLogs) (110134) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:45,863::__init__::232::IOProcess::(_processLogs) (110134) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:47,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:47,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141079 s, 72.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:47,799::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:47,801::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-80::DEBUG::2014-09-25 17:59:47,803::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:59:47,804::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:47,813::__init__::232::IOProcess::(_processLogs) (110135) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:47,815::__init__::232::IOProcess::(_processLogs) (110135) Queuing response >Thread-80::DEBUG::2014-09-25 17:59:47,818::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00160282 s, 236 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:47,819::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:47,819::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:47,819::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:47,820::__init__::232::IOProcess::(_processLogs) (110136) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:47,820::__init__::232::IOProcess::(_processLogs) (110136) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:49,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:49,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146759 s, 69.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:59:51,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:51,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.012872 s, 79.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:51,836::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:51,837::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:51,837::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:51,837::__init__::232::IOProcess::(_processLogs) (110137) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:51,838::__init__::232::IOProcess::(_processLogs) (110137) Queuing response >Thread-79::DEBUG::2014-09-25 17:59:51,839::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 17:59:51,847::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000296537 s, 1.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:51,848::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:51,849::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:51,849::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:51,849::__init__::232::IOProcess::(_processLogs) (110138) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:51,850::__init__::232::IOProcess::(_processLogs) (110138) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:51,851::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:51,851::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:51,851::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:51,852::__init__::232::IOProcess::(_processLogs) (110139) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:51,852::__init__::232::IOProcess::(_processLogs) (110139) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:51,853::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:51,853::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:51,853::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:51,854::__init__::232::IOProcess::(_processLogs) (110140) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:51,854::__init__::232::IOProcess::(_processLogs) (110140) Queuing response >Thread-29::DEBUG::2014-09-25 17:59:51,855::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:51,855::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:51,855::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:51,856::__init__::232::IOProcess::(_processLogs) (110141) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 17:59:51,856::__init__::232::IOProcess::(_processLogs) (110141) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:53,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:53,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162651 s, 63.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:55,866::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:55,866::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:55,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:55,866::__init__::232::IOProcess::(_processLogs) (110142) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:55,868::__init__::232::IOProcess::(_processLogs) (110142) Queuing response >Thread-81::DEBUG::2014-09-25 17:59:55,869::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:55,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-81::DEBUG::2014-09-25 17:59:55,885::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00282129 s, 132 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:55,887::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:55,887::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:55,887::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:55,887::__init__::232::IOProcess::(_processLogs) (110143) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:55,888::__init__::232::IOProcess::(_processLogs) (110143) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:55,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160892 s, 63.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:57,822::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:57,823::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:57,823::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 17:59:57,824::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 17:59:57,824::__init__::232::IOProcess::(_processLogs) (110144) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:57,834::__init__::232::IOProcess::(_processLogs) (110144) Queuing response >Thread-80::DEBUG::2014-09-25 17:59:57,836::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000549663 s, 688 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 17:59:57,840::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 17:59:57,841::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 17:59:57,842::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 17:59:57,844::__init__::232::IOProcess::(_processLogs) (110145) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 17:59:57,846::__init__::232::IOProcess::(_processLogs) (110145) Queuing response >Dummy-148::DEBUG::2014-09-25 17:59:57,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:57,946::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133664 s, 76.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 17:59:59,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 17:59:59,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158744 s, 64.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:01,858::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:01,858::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:01,858::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:01,859::__init__::232::IOProcess::(_processLogs) (110146) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:01,860::__init__::232::IOProcess::(_processLogs) (110146) Queuing response >Thread-79::DEBUG::2014-09-25 18:00:01,861::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:00:01,877::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00204286 s, 275 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:01,878::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:01,878::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:01,878::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:01,879::__init__::232::IOProcess::(_processLogs) (110147) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:01,879::__init__::232::IOProcess::(_processLogs) (110147) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:01,880::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:01,881::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:01,881::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:01,882::__init__::232::IOProcess::(_processLogs) (110148) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:01,882::__init__::232::IOProcess::(_processLogs) (110148) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:01,883::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:01,883::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:01,884::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:01,884::__init__::232::IOProcess::(_processLogs) (110149) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:01,884::__init__::232::IOProcess::(_processLogs) (110149) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:01,885::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:01,885::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:01,886::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:01,886::__init__::232::IOProcess::(_processLogs) (110150) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:01,886::__init__::232::IOProcess::(_processLogs) (110150) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:02,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:02,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153263 s, 66.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:04,058::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:04,090::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205593 s, 49.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:05,890::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:05,890::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:05,891::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:05,891::__init__::232::IOProcess::(_processLogs) (110151) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:05,900::__init__::232::IOProcess::(_processLogs) (110151) Queuing response >Thread-81::DEBUG::2014-09-25 18:00:05,901::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:00:05,923::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00428796 s, 86.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:05,924::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:05,925::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:05,925::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:05,925::__init__::232::IOProcess::(_processLogs) (110152) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:05,929::__init__::232::IOProcess::(_processLogs) (110152) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:06,110::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:06,157::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0331901 s, 30.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:07,849::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:07,849::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:07,850::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-80::DEBUG::2014-09-25 18:00:07,851::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:00:07,851::__init__::232::IOProcess::(_processLogs) (110153) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:07,860::__init__::232::IOProcess::(_processLogs) (110153) Queuing response >Thread-80::DEBUG::2014-09-25 18:00:07,861::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000663515 s, 570 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:07,862::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:07,862::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:07,862::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:07,863::__init__::232::IOProcess::(_processLogs) (110154) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:07,863::__init__::232::IOProcess::(_processLogs) (110154) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:08,177::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:08,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015308 s, 66.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:10,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:10,259::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207228 s, 49.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:11,887::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:11,887::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:11,887::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:11,888::__init__::232::IOProcess::(_processLogs) (110155) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:11,888::__init__::232::IOProcess::(_processLogs) (110155) Queuing response >Thread-79::DEBUG::2014-09-25 18:00:11,888::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:00:11,897::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000306101 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:11,898::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:11,899::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:11,899::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:11,899::__init__::232::IOProcess::(_processLogs) (110156) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:11,900::__init__::232::IOProcess::(_processLogs) (110156) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:11,901::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:11,902::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:11,902::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:11,902::__init__::232::IOProcess::(_processLogs) (110157) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:11,903::__init__::232::IOProcess::(_processLogs) (110157) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:11,903::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:11,903::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:11,904::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:11,904::__init__::232::IOProcess::(_processLogs) (110158) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:11,905::__init__::232::IOProcess::(_processLogs) (110158) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:11,905::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:11,906::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:11,907::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:11,907::__init__::232::IOProcess::(_processLogs) (110159) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:11,907::__init__::232::IOProcess::(_processLogs) (110159) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:12,278::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:12,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151184 s, 67.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:14,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:14,354::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165811 s, 61.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:15,931::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:15,932::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:15,932::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:15,932::__init__::232::IOProcess::(_processLogs) (110160) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:15,935::__init__::232::IOProcess::(_processLogs) (110160) Queuing response >Thread-81::DEBUG::2014-09-25 18:00:15,936::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:00:15,953::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00179084 s, 207 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:15,954::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:15,954::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:15,955::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:15,955::__init__::232::IOProcess::(_processLogs) (110161) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:15,958::__init__::232::IOProcess::(_processLogs) (110161) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:16,374::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:16,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0257759 s, 39.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:17,865::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:17,865::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:17,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:17,866::__init__::232::IOProcess::(_processLogs) (110162) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:17,866::__init__::232::IOProcess::(_processLogs) (110162) Queuing response >Thread-80::DEBUG::2014-09-25 18:00:17,868::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 18:00:17,878::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00169253 s, 223 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:17,879::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:17,879::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:17,880::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:17,881::__init__::232::IOProcess::(_processLogs) (110163) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:17,881::__init__::232::IOProcess::(_processLogs) (110163) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:18,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:18,654::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215399 s, 47.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:20,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:20,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017369 s, 59.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:21,907::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:21,908::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:21,908::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:21,908::__init__::232::IOProcess::(_processLogs) (110164) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:21,914::__init__::232::IOProcess::(_processLogs) (110164) Queuing response >Thread-79::DEBUG::2014-09-25 18:00:21,915::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:00:21,955::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00550678 s, 102 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:21,957::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:21,957::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:21,957::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:21,957::__init__::232::IOProcess::(_processLogs) (110165) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:21,959::__init__::232::IOProcess::(_processLogs) (110165) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:21,960::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:21,960::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:21,961::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:21,962::__init__::232::IOProcess::(_processLogs) (110166) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:21,962::__init__::232::IOProcess::(_processLogs) (110166) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:21,962::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:21,963::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:21,964::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:21,964::__init__::232::IOProcess::(_processLogs) (110167) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:21,964::__init__::232::IOProcess::(_processLogs) (110167) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:21,964::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:21,965::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:21,966::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:21,967::__init__::232::IOProcess::(_processLogs) (110168) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:21,967::__init__::232::IOProcess::(_processLogs) (110168) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:22,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:22,749::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163312 s, 62.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:24,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:24,791::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013337 s, 76.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:25,960::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:25,961::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:25,961::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:25,961::__init__::232::IOProcess::(_processLogs) (110169) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:25,961::__init__::232::IOProcess::(_processLogs) (110169) Queuing response >Thread-81::DEBUG::2014-09-25 18:00:25,963::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:00:25,983::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00462144 s, 80.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:25,984::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:25,985::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:25,985::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:25,985::__init__::232::IOProcess::(_processLogs) (110170) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:25,988::__init__::232::IOProcess::(_processLogs) (110170) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:26,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:26,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021081 s, 48.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:27,882::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:27,882::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:27,882::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:27,883::__init__::232::IOProcess::(_processLogs) (110171) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:27,883::__init__::232::IOProcess::(_processLogs) (110171) Queuing response >Thread-80::DEBUG::2014-09-25 18:00:27,884::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 18:00:27,894::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000569475 s, 664 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:27,895::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:27,895::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:27,896::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:27,897::__init__::232::IOProcess::(_processLogs) (110172) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:27,897::__init__::232::IOProcess::(_processLogs) (110172) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:28,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:28,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211008 s, 48.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:30,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:30,948::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156102 s, 65.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:31,967::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:31,967::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:31,967::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:31,968::__init__::232::IOProcess::(_processLogs) (110173) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:31,968::__init__::232::IOProcess::(_processLogs) (110173) Queuing response >Thread-79::DEBUG::2014-09-25 18:00:31,969::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:00:31,980::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00192097 s, 293 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:31,981::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:31,981::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:31,982::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:31,982::__init__::232::IOProcess::(_processLogs) (110174) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:31,983::__init__::232::IOProcess::(_processLogs) (110174) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:31,984::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:31,984::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:31,985::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:31,985::__init__::232::IOProcess::(_processLogs) (110175) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:31,985::__init__::232::IOProcess::(_processLogs) (110175) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:31,986::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:31,986::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:31,987::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:31,987::__init__::232::IOProcess::(_processLogs) (110176) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:31,988::__init__::232::IOProcess::(_processLogs) (110176) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:31,988::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:31,989::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:31,989::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:31,990::__init__::232::IOProcess::(_processLogs) (110177) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:31,990::__init__::232::IOProcess::(_processLogs) (110177) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:32,968::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:32,990::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133247 s, 76.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:35,010::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:35,039::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018288 s, 56.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:35,990::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:35,990::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:35,990::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:35,990::__init__::232::IOProcess::(_processLogs) (110178) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:35,992::__init__::232::IOProcess::(_processLogs) (110178) Queuing response >Thread-81::DEBUG::2014-09-25 18:00:35,992::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:00:36,009::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00588318 s, 63.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:36,010::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:36,010::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:36,010::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:36,011::__init__::232::IOProcess::(_processLogs) (110179) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:36,017::__init__::232::IOProcess::(_processLogs) (110179) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:37,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:37,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134905 s, 75.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:37,898::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:37,898::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:37,899::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:37,899::__init__::232::IOProcess::(_processLogs) (110180) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:37,899::__init__::232::IOProcess::(_processLogs) (110180) Queuing response >Thread-80::DEBUG::2014-09-25 18:00:37,901::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 18:00:37,910::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.00053844 s, 702 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:37,911::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:37,911::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:37,911::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:37,912::__init__::232::IOProcess::(_processLogs) (110181) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:37,912::__init__::232::IOProcess::(_processLogs) (110181) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:39,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:39,134::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218927 s, 46.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:41,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:41,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189279 s, 54.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:41,990::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:41,991::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:41,991::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:41,991::__init__::232::IOProcess::(_processLogs) (110182) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:41,996::__init__::232::IOProcess::(_processLogs) (110182) Queuing response >Thread-79::DEBUG::2014-09-25 18:00:41,997::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:00:42,034::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0173931 s, 32.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:42,035::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:42,036::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:42,036::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:42,036::__init__::232::IOProcess::(_processLogs) (110183) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:42,041::__init__::232::IOProcess::(_processLogs) (110183) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:42,043::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:42,043::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:42,044::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:42,044::__init__::232::IOProcess::(_processLogs) (110184) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:42,044::__init__::232::IOProcess::(_processLogs) (110184) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:42,045::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:42,045::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:42,045::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:42,045::__init__::232::IOProcess::(_processLogs) (110185) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:42,046::__init__::232::IOProcess::(_processLogs) (110185) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:42,046::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:42,047::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:42,047::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:42,047::__init__::232::IOProcess::(_processLogs) (110186) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:42,048::__init__::232::IOProcess::(_processLogs) (110186) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:43,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:43,231::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156947 s, 65.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:45,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:45,284::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189081 s, 54.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:46,020::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:46,021::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:46,022::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:46,023::__init__::232::IOProcess::(_processLogs) (110187) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:46,025::__init__::232::IOProcess::(_processLogs) (110187) Queuing response >Thread-81::DEBUG::2014-09-25 18:00:46,027::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:00:46,044::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00407186 s, 91.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:46,047::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:46,048::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:46,050::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:46,052::__init__::232::IOProcess::(_processLogs) (110188) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:46,053::__init__::232::IOProcess::(_processLogs) (110188) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:47,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:47,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180757 s, 56.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:47,914::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:47,915::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:47,915::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:47,915::__init__::232::IOProcess::(_processLogs) (110189) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:47,916::__init__::232::IOProcess::(_processLogs) (110189) Queuing response >Thread-80::DEBUG::2014-09-25 18:00:47,916::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 18:00:47,925::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000414003 s, 913 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:47,927::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:47,927::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:47,927::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:47,929::__init__::232::IOProcess::(_processLogs) (110190) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:47,929::__init__::232::IOProcess::(_processLogs) (110190) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:49,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:49,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162389 s, 63.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:51,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:51,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160171 s, 63.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:52,049::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:52,049::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:52,050::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:52,050::__init__::232::IOProcess::(_processLogs) (110191) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:52,053::__init__::232::IOProcess::(_processLogs) (110191) Queuing response >Thread-79::DEBUG::2014-09-25 18:00:52,054::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:00:52,070::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0042635 s, 132 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:52,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:52,072::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:52,072::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:52,072::__init__::232::IOProcess::(_processLogs) (110192) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:52,076::__init__::232::IOProcess::(_processLogs) (110192) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:52,077::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:52,078::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:52,078::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:52,078::__init__::232::IOProcess::(_processLogs) (110193) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:52,080::__init__::232::IOProcess::(_processLogs) (110193) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:52,081::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:52,081::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:52,081::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:52,081::__init__::232::IOProcess::(_processLogs) (110194) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:52,085::__init__::232::IOProcess::(_processLogs) (110194) Queuing response >Thread-29::DEBUG::2014-09-25 18:00:52,086::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:52,086::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:52,086::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:52,086::__init__::232::IOProcess::(_processLogs) (110195) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:00:52,088::__init__::232::IOProcess::(_processLogs) (110195) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:53,450::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:53,480::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175191 s, 58.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:00:55,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:55,539::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0247627 s, 41.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:56,053::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:56,053::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:56,053::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:56,054::__init__::232::IOProcess::(_processLogs) (110196) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:56,061::__init__::232::IOProcess::(_processLogs) (110196) Queuing response >Thread-81::DEBUG::2014-09-25 18:00:56,062::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:00:56,079::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00730673 s, 50.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:56,080::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:56,081::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:56,081::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:56,081::__init__::232::IOProcess::(_processLogs) (110197) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:56,086::__init__::232::IOProcess::(_processLogs) (110197) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:57,559::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:57,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142882 s, 71.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:57,930::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:57,930::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:57,930::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:57,931::__init__::232::IOProcess::(_processLogs) (110198) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:57,931::__init__::232::IOProcess::(_processLogs) (110198) Queuing response >Thread-80::DEBUG::2014-09-25 18:00:57,932::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 18:00:57,941::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000453976 s, 833 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:00:57,942::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:00:57,943::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:00:57,943::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:00:57,943::__init__::232::IOProcess::(_processLogs) (110199) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:00:57,944::__init__::232::IOProcess::(_processLogs) (110199) Queuing response >Dummy-148::DEBUG::2014-09-25 18:00:59,604::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:00:59,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0232457 s, 44.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:01,656::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:01,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132944 s, 77.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:02,090::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:02,091::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:02,091::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:02,091::__init__::232::IOProcess::(_processLogs) (110200) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:02,097::__init__::232::IOProcess::(_processLogs) (110200) Queuing response >Thread-79::DEBUG::2014-09-25 18:01:02,098::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:01:02,113::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00542081 s, 104 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:02,114::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:02,115::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:02,115::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:02,115::__init__::232::IOProcess::(_processLogs) (110201) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:02,117::__init__::232::IOProcess::(_processLogs) (110201) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:02,118::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:02,119::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:02,119::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:02,120::__init__::232::IOProcess::(_processLogs) (110202) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:02,120::__init__::232::IOProcess::(_processLogs) (110202) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:02,120::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:02,121::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:02,121::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:02,122::__init__::232::IOProcess::(_processLogs) (110203) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:02,122::__init__::232::IOProcess::(_processLogs) (110203) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:02,122::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:02,123::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:02,123::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:02,124::__init__::232::IOProcess::(_processLogs) (110204) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:02,124::__init__::232::IOProcess::(_processLogs) (110204) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:03,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:03,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0196007 s, 52.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:05,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:06,036::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.268451 s, 3.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:06,087::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:06,088::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:06,088::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:06,088::__init__::232::IOProcess::(_processLogs) (110205) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:06,097::__init__::232::IOProcess::(_processLogs) (110205) Queuing response >Thread-81::DEBUG::2014-09-25 18:01:06,098::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:01:06,125::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00633939 s, 58.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:06,126::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:06,126::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:06,127::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:06,127::__init__::232::IOProcess::(_processLogs) (110206) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:06,130::__init__::232::IOProcess::(_processLogs) (110206) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:07,946::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:07,946::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:07,946::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:07,947::__init__::232::IOProcess::(_processLogs) (110207) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:07,947::__init__::232::IOProcess::(_processLogs) (110207) Queuing response >Thread-80::DEBUG::2014-09-25 18:01:07,948::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 18:01:07,957::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000383492 s, 986 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:07,959::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:07,959::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:07,959::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:07,959::__init__::232::IOProcess::(_processLogs) (110208) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:07,960::__init__::232::IOProcess::(_processLogs) (110208) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:08,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:08,083::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157663 s, 64.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:10,104::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:10,126::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131979 s, 77.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:12,125::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:12,125::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:12,126::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:12,126::__init__::232::IOProcess::(_processLogs) (110209) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:12,126::__init__::232::IOProcess::(_processLogs) (110209) Queuing response >Thread-79::DEBUG::2014-09-25 18:01:12,127::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:01:12,136::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000317587 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:12,137::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:12,138::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:12,139::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:12,140::__init__::232::IOProcess::(_processLogs) (110210) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:12,140::__init__::232::IOProcess::(_processLogs) (110210) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:12,140::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:12,141::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:12,141::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:12,142::__init__::232::IOProcess::(_processLogs) (110211) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:12,142::__init__::232::IOProcess::(_processLogs) (110211) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:12,143::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:12,143::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:12,144::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:12,144::__init__::232::IOProcess::(_processLogs) (110212) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:12,144::__init__::232::IOProcess::(_processLogs) (110212) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:12,145::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:12,145::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:12,147::__init__::232::IOProcess::(_processLogs) Extracting request information... >Dummy-148::DEBUG::2014-09-25 18:01:12,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 18:01:12,148::__init__::232::IOProcess::(_processLogs) (110213) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:12,156::__init__::232::IOProcess::(_processLogs) (110213) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:12,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133503 s, 76.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:14,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:14,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212139 s, 48.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:16,134::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:16,135::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:16,137::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:16,138::__init__::232::IOProcess::(_processLogs) (110214) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:16,139::__init__::232::IOProcess::(_processLogs) (110214) Queuing response >Thread-81::DEBUG::2014-09-25 18:01:16,141::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:01:16,152::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000280293 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:16,155::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:16,156::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:16,157::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:16,158::__init__::232::IOProcess::(_processLogs) (110215) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:16,160::__init__::232::IOProcess::(_processLogs) (110215) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:16,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:16,279::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0221765 s, 46.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:17,962::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:17,963::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:17,963::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:17,963::__init__::232::IOProcess::(_processLogs) (110216) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:17,964::__init__::232::IOProcess::(_processLogs) (110216) Queuing response >Thread-80::DEBUG::2014-09-25 18:01:17,965::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-80::DEBUG::2014-09-25 18:01:17,974::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n378 bytes (378 B) copied, 0.000449597 s, 841 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:17,975::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:17,975::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:17,976::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:17,976::__init__::232::IOProcess::(_processLogs) (110217) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:17,977::__init__::232::IOProcess::(_processLogs) (110217) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:18,472::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:18,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192159 s, 53.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:20,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:20,547::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131466 s, 77.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:22,148::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:22,148::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:22,148::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:22,149::__init__::232::IOProcess::(_processLogs) (110218) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:22,168::__init__::232::IOProcess::(_processLogs) (110218) Queuing response >Thread-79::DEBUG::2014-09-25 18:01:22,169::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:01:22,178::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000386895 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:22,180::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:22,180::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:22,181::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:22,182::__init__::232::IOProcess::(_processLogs) (110219) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:22,182::__init__::232::IOProcess::(_processLogs) (110219) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:22,182::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:22,183::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:22,183::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:22,184::__init__::232::IOProcess::(_processLogs) (110220) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:22,184::__init__::232::IOProcess::(_processLogs) (110220) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:22,184::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:22,185::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:22,185::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:22,186::__init__::232::IOProcess::(_processLogs) (110221) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:22,186::__init__::232::IOProcess::(_processLogs) (110221) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:22,186::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:22,187::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:22,187::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:22,188::__init__::232::IOProcess::(_processLogs) (110222) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:22,188::__init__::232::IOProcess::(_processLogs) (110222) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:22,567::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:22,606::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0285928 s, 35.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:24,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:24,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130593 s, 78.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:26,163::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:26,164::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:26,164::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:26,164::__init__::232::IOProcess::(_processLogs) (110223) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:26,165::__init__::232::IOProcess::(_processLogs) (110223) Queuing response >Thread-81::DEBUG::2014-09-25 18:01:26,166::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:01:26,194::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0110103 s, 33.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:26,195::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:26,195::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:26,195::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:26,196::__init__::232::IOProcess::(_processLogs) (110224) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:26,202::__init__::232::IOProcess::(_processLogs) (110224) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:26,667::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:26,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173683 s, 59.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:27,979::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:27,979::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:27,979::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:27,980::__init__::232::IOProcess::(_processLogs) (110225) Got request for method 'statvfs' >Dummy-148::DEBUG::2014-09-25 18:01:28,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:28,767::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0267644 s, 38.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:30,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:30,809::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133015 s, 77.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:32,189::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:32,189::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:32,190::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:32,190::__init__::232::IOProcess::(_processLogs) (110226) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:32,191::__init__::232::IOProcess::(_processLogs) (110226) Queuing response >Thread-79::DEBUG::2014-09-25 18:01:32,192::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:01:32,207::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00459101 s, 122 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:32,208::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:32,208::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:32,209::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:32,209::__init__::232::IOProcess::(_processLogs) (110227) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:32,210::__init__::232::IOProcess::(_processLogs) (110227) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:32,212::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:32,212::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:32,213::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:32,213::__init__::232::IOProcess::(_processLogs) (110228) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:32,213::__init__::232::IOProcess::(_processLogs) (110228) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:32,214::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:32,214::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:32,214::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:32,214::__init__::232::IOProcess::(_processLogs) (110229) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:32,215::__init__::232::IOProcess::(_processLogs) (110229) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:32,215::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:32,216::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:32,216::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:32,216::__init__::232::IOProcess::(_processLogs) (110230) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:32,216::__init__::232::IOProcess::(_processLogs) (110230) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:32,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:32,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169768 s, 60.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:34,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:34,909::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193474 s, 52.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:36,204::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:36,204::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:36,204::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:36,205::__init__::232::IOProcess::(_processLogs) (110231) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:36,209::__init__::232::IOProcess::(_processLogs) (110231) Queuing response >Thread-81::DEBUG::2014-09-25 18:01:36,210::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:01:36,226::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0051817 s, 71.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:36,228::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:36,228::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:36,228::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:36,228::__init__::232::IOProcess::(_processLogs) (110232) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:36,234::__init__::232::IOProcess::(_processLogs) (110232) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:36,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:36,993::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0293663 s, 34.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:39,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:39,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154881 s, 66.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:41,061::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:41,089::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147449 s, 69.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:42,218::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:42,218::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:42,218::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:42,219::__init__::232::IOProcess::(_processLogs) (110233) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:42,220::__init__::232::IOProcess::(_processLogs) (110233) Queuing response >Thread-79::DEBUG::2014-09-25 18:01:42,221::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:01:42,235::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00171971 s, 327 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:42,236::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:42,236::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:42,237::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:42,237::__init__::232::IOProcess::(_processLogs) (110234) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:42,242::__init__::232::IOProcess::(_processLogs) (110234) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:42,243::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:42,244::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:42,245::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:42,245::__init__::232::IOProcess::(_processLogs) (110235) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:42,245::__init__::232::IOProcess::(_processLogs) (110235) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:42,246::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:42,246::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:42,247::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:42,247::__init__::232::IOProcess::(_processLogs) (110236) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:42,247::__init__::232::IOProcess::(_processLogs) (110236) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:42,248::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:42,248::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:42,249::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:42,250::__init__::232::IOProcess::(_processLogs) (110237) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:42,250::__init__::232::IOProcess::(_processLogs) (110237) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:43,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:43,139::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166105 s, 61.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:45,159::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:45,193::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179051 s, 57.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:46,237::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:46,238::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:46,239::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:46,241::__init__::232::IOProcess::(_processLogs) (110238) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:46,242::__init__::232::IOProcess::(_processLogs) (110238) Queuing response >Thread-81::DEBUG::2014-09-25 18:01:46,245::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:01:46,255::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000350587 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:46,258::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:46,259::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:46,260::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:46,261::__init__::232::IOProcess::(_processLogs) (110239) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:46,264::__init__::232::IOProcess::(_processLogs) (110239) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:47,213::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:47,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133024 s, 77.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:01:49,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:49,289::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192436 s, 53.2 MB/s\n'; <rc> = 0 >Detector thread::DEBUG::2014-09-25 18:01:50,567::protocoldetector::160::vds.MultiProtocolAcceptor::(_add_connection) Adding connection from 10.34.63.74:54203 >Detector thread::DEBUG::2014-09-25 18:01:50,575::protocoldetector::171::vds.MultiProtocolAcceptor::(_remove_connection) Connection removed from 10.34.63.74:54203 >Detector thread::DEBUG::2014-09-25 18:01:50,575::protocoldetector::197::vds.MultiProtocolAcceptor::(_handle_connection_read) Detected protocol stomp from 10.34.63.74:54203 >Detector thread::DEBUG::2014-09-25 18:01:50,576::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='CONNECT'> >Detector thread::INFO::2014-09-25 18:01:50,576::stompReactor::66::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request >Detector thread::DEBUG::2014-09-25 18:01:50,577::stompReactor::338::protocoldetector.StompDetector::(handleSocket) Stomp detected from ('10.34.63.74', 54203) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:50,578::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SUBSCRIBE'> >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:50,578::stompReactor::89::Broker.StompAdapter::(_cmd_subscribe) Subscribe command ignored >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:50,618::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:01:50,619::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71575::DEBUG::2014-09-25 18:01:50,622::task::595::Storage.TaskManager.Task::(_updateState) Task=`74736684-052e-47b0-a0c8-ccd458d18994`::moving from state init -> state preparing >Thread-71575::INFO::2014-09-25 18:01:50,622::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71575::INFO::2014-09-25 18:01:50,623::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00171971', 'lastCheck': '8.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449597', 'lastCheck': '32.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000350587', 'lastCheck': '4.4', 'valid': True}} >Thread-71575::DEBUG::2014-09-25 18:01:50,623::task::1191::Storage.TaskManager.Task::(prepare) Task=`74736684-052e-47b0-a0c8-ccd458d18994`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00171971', 'lastCheck': '8.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449597', 'lastCheck': '32.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000350587', 'lastCheck': '4.4', 'valid': True}} >Thread-71575::DEBUG::2014-09-25 18:01:50,623::task::595::Storage.TaskManager.Task::(_updateState) Task=`74736684-052e-47b0-a0c8-ccd458d18994`::moving from state preparing -> state finished >Thread-71575::DEBUG::2014-09-25 18:01:50,623::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71575::DEBUG::2014-09-25 18:01:50,624::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71575::DEBUG::2014-09-25 18:01:50,624::task::993::Storage.TaskManager.Task::(_decref) Task=`74736684-052e-47b0-a0c8-ccd458d18994`::ref 0 aborting False >Thread-71575::DEBUG::2014-09-25 18:01:50,626::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:50,788::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71576::DEBUG::2014-09-25 18:01:50,789::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getCapabilities' in bridge with {} >JsonRpcServer::DEBUG::2014-09-25 18:01:50,789::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71576::DEBUG::2014-09-25 18:01:50,793::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) >Thread-71576::DEBUG::2014-09-25 18:01:50,803::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-71576::DEBUG::2014-09-25 18:01:50,840::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getCapabilities' in bridge with {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:3bd6acb1d4f2'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.el7.x86_64', 'buildtime': 1399281417.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1391017388L, 'version': '0.12.4'}, 'vdsm': {'release': '3.el7.beta', 'buildtime': 1410353965L, 'version': '4.16.3'}, 'qemu-kvm': {'release': '60.el7_0.7', 'buildtime': 1409068381L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.1', 'buildtime': 1404387754L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.7', 'buildtime': 1409068381L, 'version': '1.5.3'}, 'mom': {'release': '1.el7_0', 'buildtime': 1405326698L, 'version': '0.4.1'}}, 'numaNodeDistance': {'0': [10]}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5645 @ 2.40GHz', 'liveMerge': 'false', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'selinux': {'mode': '1'}, 'kdumpStatus': 0, 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.63.137', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': ['2620:52:0:223c:868f:69ff:fe67:1f0f/64', 'fe80::868f:69ff:fe67:1f0f/64'], 'gateway': '10.34.63.254', 'bootproto4': 'dhcp', 'netmask': '255.255.252.0', 'stp': 'off', 'ipv4addrs': ['10.34.63.137/22'], 'mtu': '1500', 'ipv6gateway': 'fe80:52:0:223c::3fe', 'ports': ['em1']}}, 'bridges': {'rhevm': {'addr': '10.34.63.137', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['2620:52:0:223c:868f:69ff:fe67:1f0f/64', 'fe80::868f:69ff:fe67:1f0f/64'], 'mtu': '1500', 'netmask': '255.255.252.0', 'stp': 'off', 'ipv4addrs': ['10.34.63.137/22'], 'ipv6gateway': 'fe80:52:0:223c::3fe', 'gateway': '10.34.63.254', 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '65', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.848f69671f0f', 'bridge_id': '8000.848f69671f0f', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '5968', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'multicast_query_interval': '12500', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}, 'ports': ['em1']}}, 'uuid': '4C4C4544-0052-4C10-8058-B5C04F43354A', 'lastClientIface': 'rhevm', 'nics': {'em1': {'addr': '', 'cfg': {'BRIDGE': 'rhevm', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '84:8f:69:67:1f:0f', 'DEVICE': 'em1', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::868f:69ff:fe67:1f0f/64'], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '84:8f:69:67:1f:0f', 'speed': 1000}, 'em2': {'addr': '', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em2', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '6d707312-7e57-4ec0-ba12-7cb025979f88', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': '84:8F:69:67:1F:11', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '84:8f:69:67:1f:11', 'speed': 0}}, 'software_revision': '3', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,ida,arat,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:3bd6acb1d4f2', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'], 'autoNumaBalancing': 0, 'reservedMem': '321', 'bondings': {'bond0': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '76:ee:22:47:c3:e4', 'slaves': [], 'opts': {}}}, 'software_version': '4.16', 'memSize': '19903', 'cpuSpeed': '2393.935', 'numaNodes': {u'0': {'totalMemory': '19903', 'cpus': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}}, 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '6', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '12', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '1.el7', 'version': '7.0', 'name': 'RHEL'}} >Thread-71576::DEBUG::2014-09-25 18:01:50,841::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:50,882::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71577::DEBUG::2014-09-25 18:01:50,883::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getHardwareInfo' in bridge with {} >JsonRpcServer::DEBUG::2014-09-25 18:01:50,883::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71577::DEBUG::2014-09-25 18:01:50,887::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getHardwareInfo' in bridge with {'systemProductName': 'PowerEdge M610', 'systemUUID': '4C4C4544-0052-4C10-8058-B5C04F43354A', 'systemSerialNumber': '5RLXC5J', 'systemManufacturer': 'Dell Inc.'} >Thread-71577::DEBUG::2014-09-25 18:01:50,888::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:50,966::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:01:50,967::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71578::DEBUG::2014-09-25 18:01:50,969::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:01:51,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:51,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209172 s, 49.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:52,250::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:52,250::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:52,251::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:52,251::__init__::232::IOProcess::(_processLogs) (110240) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:52,255::__init__::232::IOProcess::(_processLogs) (110240) Queuing response >Thread-79::DEBUG::2014-09-25 18:01:52,256::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:01:52,282::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0147558 s, 38.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:52,283::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:52,283::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:52,284::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:52,284::__init__::232::IOProcess::(_processLogs) (110241) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:52,287::__init__::232::IOProcess::(_processLogs) (110241) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:52,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:52,288::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:52,289::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:52,289::__init__::232::IOProcess::(_processLogs) (110242) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:52,291::__init__::232::IOProcess::(_processLogs) (110242) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:52,292::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:52,292::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:52,293::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:52,293::__init__::232::IOProcess::(_processLogs) (110243) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:52,297::__init__::232::IOProcess::(_processLogs) (110243) Queuing response >Thread-29::DEBUG::2014-09-25 18:01:52,298::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:52,299::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:52,299::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:52,299::__init__::232::IOProcess::(_processLogs) (110244) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:01:52,302::__init__::232::IOProcess::(_processLogs) (110244) Queuing response >Dummy-148::DEBUG::2014-09-25 18:01:53,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:53,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155498 s, 65.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:54,409::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:01:54,410::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71579::DEBUG::2014-09-25 18:01:54,412::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:01:55,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:55,456::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0283258 s, 36.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:56,267::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:56,267::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:56,267::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:56,268::__init__::232::IOProcess::(_processLogs) (110245) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:56,271::__init__::232::IOProcess::(_processLogs) (110245) Queuing response >Thread-81::DEBUG::2014-09-25 18:01:56,272::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:01:56,290::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00585617 s, 63.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:01:56,291::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:01:56,291::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:01:56,292::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:01:56,292::__init__::232::IOProcess::(_processLogs) (110246) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:01:56,294::__init__::232::IOProcess::(_processLogs) (110246) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:57,436::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:01:57,438::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71580::DEBUG::2014-09-25 18:01:57,439::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:01:57,476::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:57,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135023 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:58,007::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:01:58,008::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71581::DEBUG::2014-09-25 18:01:58,008::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71581::DEBUG::2014-09-25 18:01:58,010::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ebbe8c0-341f-49e0-818f-e51cb11ffd60`::moving from state init -> state preparing >Thread-71581::INFO::2014-09-25 18:01:58,011::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71581::INFO::2014-09-25 18:01:58,011::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71581::DEBUG::2014-09-25 18:01:58,011::task::1191::Storage.TaskManager.Task::(prepare) Task=`4ebbe8c0-341f-49e0-818f-e51cb11ffd60`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71581::DEBUG::2014-09-25 18:01:58,012::task::595::Storage.TaskManager.Task::(_updateState) Task=`4ebbe8c0-341f-49e0-818f-e51cb11ffd60`::moving from state preparing -> state finished >Thread-71581::DEBUG::2014-09-25 18:01:58,012::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71581::DEBUG::2014-09-25 18:01:58,012::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71581::DEBUG::2014-09-25 18:01:58,012::task::993::Storage.TaskManager.Task::(_decref) Task=`4ebbe8c0-341f-49e0-818f-e51cb11ffd60`::ref 0 aborting False >Thread-71581::DEBUG::2014-09-25 18:01:58,013::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71581::DEBUG::2014-09-25 18:01:58,013::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Detector thread::DEBUG::2014-09-25 18:01:58,117::protocoldetector::160::vds.MultiProtocolAcceptor::(_add_connection) Adding connection from 10.34.63.74:54204 >Detector thread::DEBUG::2014-09-25 18:01:58,119::protocoldetector::171::vds.MultiProtocolAcceptor::(_remove_connection) Connection removed from 10.34.63.74:54204 >Detector thread::DEBUG::2014-09-25 18:01:58,119::protocoldetector::197::vds.MultiProtocolAcceptor::(_handle_connection_read) Detected protocol stomp from 10.34.63.74:54204 >Detector thread::DEBUG::2014-09-25 18:01:58,120::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='CONNECT'> >Detector thread::INFO::2014-09-25 18:01:58,120::stompReactor::66::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request >Detector thread::DEBUG::2014-09-25 18:01:58,120::stompReactor::338::protocoldetector.StompDetector::(handleSocket) Stomp detected from ('10.34.63.74', 54204) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:58,122::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SUBSCRIBE'> >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:58,122::stompReactor::89::Broker.StompAdapter::(_cmd_subscribe) Subscribe command ignored >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:58,162::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71582::DEBUG::2014-09-25 18:01:58,163::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:01:58,163::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71582::DEBUG::2014-09-25 18:01:58,165::task::595::Storage.TaskManager.Task::(_updateState) Task=`5c27e6fa-fcc4-4bec-b51a-dbc6bf716a9e`::moving from state init -> state preparing >Thread-71582::INFO::2014-09-25 18:01:58,165::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71582::DEBUG::2014-09-25 18:01:58,166::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9e7f77d3-c144-46a1-83f3-47a6567282bd`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71582::DEBUG::2014-09-25 18:01:58,166::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71582::DEBUG::2014-09-25 18:01:58,166::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71582::DEBUG::2014-09-25 18:01:58,167::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9e7f77d3-c144-46a1-83f3-47a6567282bd`::Granted request >Thread-71582::DEBUG::2014-09-25 18:01:58,167::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`5c27e6fa-fcc4-4bec-b51a-dbc6bf716a9e`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71582::DEBUG::2014-09-25 18:01:58,167::task::993::Storage.TaskManager.Task::(_decref) Task=`5c27e6fa-fcc4-4bec-b51a-dbc6bf716a9e`::ref 1 aborting False >Thread-71582::INFO::2014-09-25 18:01:58,172::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '6831472640', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71582::DEBUG::2014-09-25 18:01:58,172::task::1191::Storage.TaskManager.Task::(prepare) Task=`5c27e6fa-fcc4-4bec-b51a-dbc6bf716a9e`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '6831472640', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71582::DEBUG::2014-09-25 18:01:58,172::task::595::Storage.TaskManager.Task::(_updateState) Task=`5c27e6fa-fcc4-4bec-b51a-dbc6bf716a9e`::moving from state preparing -> state finished >Thread-71582::DEBUG::2014-09-25 18:01:58,172::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71582::DEBUG::2014-09-25 18:01:58,173::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71582::DEBUG::2014-09-25 18:01:58,173::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71582::DEBUG::2014-09-25 18:01:58,173::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71582::DEBUG::2014-09-25 18:01:58,174::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71582::DEBUG::2014-09-25 18:01:58,174::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71582::DEBUG::2014-09-25 18:01:58,174::task::993::Storage.TaskManager.Task::(_decref) Task=`5c27e6fa-fcc4-4bec-b51a-dbc6bf716a9e`::ref 0 aborting False >Thread-71582::DEBUG::2014-09-25 18:01:58,175::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '6831472640', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71582::DEBUG::2014-09-25 18:01:58,175::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:01:58,244::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:01:58,245::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71583::DEBUG::2014-09-25 18:01:58,245::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getAllTasksInfo' in bridge with {} >Thread-71583::DEBUG::2014-09-25 18:01:58,247::task::595::Storage.TaskManager.Task::(_updateState) Task=`043a8c60-dc76-4bd3-adde-f9602e65c9d8`::moving from state init -> state preparing >Thread-71583::INFO::2014-09-25 18:01:58,247::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksInfo(spUUID=None, options=None) >Thread-71583::DEBUG::2014-09-25 18:01:58,247::taskManager::191::Storage.TaskManager::(getAllTasksInfo) Entry. >Thread-71583::DEBUG::2014-09-25 18:01:58,248::taskManager::202::Storage.TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-71583::INFO::2014-09-25 18:01:58,248::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksInfo, Return response: {'allTasksInfo': {}} >Thread-71583::DEBUG::2014-09-25 18:01:58,248::task::1191::Storage.TaskManager.Task::(prepare) Task=`043a8c60-dc76-4bd3-adde-f9602e65c9d8`::finished: {'allTasksInfo': {}} >Thread-71583::DEBUG::2014-09-25 18:01:58,248::task::595::Storage.TaskManager.Task::(_updateState) Task=`043a8c60-dc76-4bd3-adde-f9602e65c9d8`::moving from state preparing -> state finished >Thread-71583::DEBUG::2014-09-25 18:01:58,249::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71583::DEBUG::2014-09-25 18:01:58,249::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71583::DEBUG::2014-09-25 18:01:58,249::task::993::Storage.TaskManager.Task::(_decref) Task=`043a8c60-dc76-4bd3-adde-f9602e65c9d8`::ref 0 aborting False >Thread-71583::DEBUG::2014-09-25 18:01:58,249::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getAllTasksInfo' in bridge with {} >Thread-71583::DEBUG::2014-09-25 18:01:58,250::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:01:59,518::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:01:59,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173467 s, 59.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:00,483::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:00,484::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71584::DEBUG::2014-09-25 18:02:00,486::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:01,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:01,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0256344 s, 39.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:02,304::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:02,304::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:02,304::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:02,305::__init__::232::IOProcess::(_processLogs) (110247) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:02,306::__init__::232::IOProcess::(_processLogs) (110247) Queuing response >Thread-79::DEBUG::2014-09-25 18:02:02,306::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:02:02,322::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00506526 s, 111 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:02,323::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:02,323::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:02,323::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:02,324::__init__::232::IOProcess::(_processLogs) (110248) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:02,325::__init__::232::IOProcess::(_processLogs) (110248) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:02,327::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:02,327::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:02,328::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:02,328::__init__::232::IOProcess::(_processLogs) (110249) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:02,328::__init__::232::IOProcess::(_processLogs) (110249) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:02,329::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:02,329::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:02,330::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:02,330::__init__::232::IOProcess::(_processLogs) (110250) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:02,331::__init__::232::IOProcess::(_processLogs) (110250) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:02,331::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:02,332::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:02,333::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:02,333::__init__::232::IOProcess::(_processLogs) (110251) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:02,333::__init__::232::IOProcess::(_processLogs) (110251) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:03,515::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:03,516::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71585::DEBUG::2014-09-25 18:02:03,518::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:03,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:03,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.03838 s, 26.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:02:05,703::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:05,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0237518 s, 43.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:06,296::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:06,297::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:06,297::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:06,297::__init__::232::IOProcess::(_processLogs) (110252) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:06,304::__init__::232::IOProcess::(_processLogs) (110252) Queuing response >Thread-81::DEBUG::2014-09-25 18:02:06,305::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:02:06,316::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000447053 s, 830 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:06,318::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:06,318::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:06,318::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:06,318::__init__::232::IOProcess::(_processLogs) (110253) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:06,319::__init__::232::IOProcess::(_processLogs) (110253) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:06,536::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:06,537::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71586::DEBUG::2014-09-25 18:02:06,540::task::595::Storage.TaskManager.Task::(_updateState) Task=`10cdbcd5-4af9-4dcc-ac54-75f132ff3bd4`::moving from state init -> state preparing >Thread-71586::INFO::2014-09-25 18:02:06,540::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71586::INFO::2014-09-25 18:02:06,540::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00506526', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449597', 'lastCheck': '48.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000447053', 'lastCheck': '0.2', 'valid': True}} >Thread-71586::DEBUG::2014-09-25 18:02:06,541::task::1191::Storage.TaskManager.Task::(prepare) Task=`10cdbcd5-4af9-4dcc-ac54-75f132ff3bd4`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00506526', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449597', 'lastCheck': '48.6', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000447053', 'lastCheck': '0.2', 'valid': True}} >Thread-71586::DEBUG::2014-09-25 18:02:06,541::task::595::Storage.TaskManager.Task::(_updateState) Task=`10cdbcd5-4af9-4dcc-ac54-75f132ff3bd4`::moving from state preparing -> state finished >Thread-71586::DEBUG::2014-09-25 18:02:06,541::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71586::DEBUG::2014-09-25 18:02:06,541::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71586::DEBUG::2014-09-25 18:02:06,542::task::993::Storage.TaskManager.Task::(_decref) Task=`10cdbcd5-4af9-4dcc-ac54-75f132ff3bd4`::ref 0 aborting False >Thread-71586::DEBUG::2014-09-25 18:02:06,544::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:06,568::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:06,569::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71587::DEBUG::2014-09-25 18:02:06,570::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:07,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:07,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148297 s, 69.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:08,244::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71588::DEBUG::2014-09-25 18:02:08,246::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:02:08,246::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71588::DEBUG::2014-09-25 18:02:08,248::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5f10625-6dbf-4999-a40d-a9423e03f614`::moving from state init -> state preparing >Thread-71588::INFO::2014-09-25 18:02:08,248::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71588::INFO::2014-09-25 18:02:08,249::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71588::DEBUG::2014-09-25 18:02:08,249::task::1191::Storage.TaskManager.Task::(prepare) Task=`a5f10625-6dbf-4999-a40d-a9423e03f614`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71588::DEBUG::2014-09-25 18:02:08,249::task::595::Storage.TaskManager.Task::(_updateState) Task=`a5f10625-6dbf-4999-a40d-a9423e03f614`::moving from state preparing -> state finished >Thread-71588::DEBUG::2014-09-25 18:02:08,250::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71588::DEBUG::2014-09-25 18:02:08,250::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71588::DEBUG::2014-09-25 18:02:08,250::task::993::Storage.TaskManager.Task::(_decref) Task=`a5f10625-6dbf-4999-a40d-a9423e03f614`::ref 0 aborting False >Thread-71588::DEBUG::2014-09-25 18:02:08,250::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71588::DEBUG::2014-09-25 18:02:08,251::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:08,266::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:08,267::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71589::DEBUG::2014-09-25 18:02:08,267::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71589::DEBUG::2014-09-25 18:02:08,269::task::595::Storage.TaskManager.Task::(_updateState) Task=`fed8094d-83bf-495c-82a4-f98fa83cc3a6`::moving from state init -> state preparing >Thread-71589::INFO::2014-09-25 18:02:08,269::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71589::DEBUG::2014-09-25 18:02:08,270::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9fa834a0-e3b2-4e62-8f6e-8e33f3523aab`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71589::DEBUG::2014-09-25 18:02:08,270::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71589::DEBUG::2014-09-25 18:02:08,270::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71589::DEBUG::2014-09-25 18:02:08,271::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`9fa834a0-e3b2-4e62-8f6e-8e33f3523aab`::Granted request >Thread-71589::DEBUG::2014-09-25 18:02:08,271::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`fed8094d-83bf-495c-82a4-f98fa83cc3a6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71589::DEBUG::2014-09-25 18:02:08,271::task::993::Storage.TaskManager.Task::(_decref) Task=`fed8094d-83bf-495c-82a4-f98fa83cc3a6`::ref 1 aborting False >Thread-71589::INFO::2014-09-25 18:02:08,276::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '6831472640', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71589::DEBUG::2014-09-25 18:02:08,276::task::1191::Storage.TaskManager.Task::(prepare) Task=`fed8094d-83bf-495c-82a4-f98fa83cc3a6`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '6831472640', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71589::DEBUG::2014-09-25 18:02:08,276::task::595::Storage.TaskManager.Task::(_updateState) Task=`fed8094d-83bf-495c-82a4-f98fa83cc3a6`::moving from state preparing -> state finished >Thread-71589::DEBUG::2014-09-25 18:02:08,277::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71589::DEBUG::2014-09-25 18:02:08,277::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71589::DEBUG::2014-09-25 18:02:08,277::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71589::DEBUG::2014-09-25 18:02:08,278::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71589::DEBUG::2014-09-25 18:02:08,278::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71589::DEBUG::2014-09-25 18:02:08,278::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71589::DEBUG::2014-09-25 18:02:08,278::task::993::Storage.TaskManager.Task::(_decref) Task=`fed8094d-83bf-495c-82a4-f98fa83cc3a6`::ref 0 aborting False >Thread-71589::DEBUG::2014-09-25 18:02:08,279::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '6831472640', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71589::DEBUG::2014-09-25 18:02:08,279::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:09,637::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:09,638::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71590::DEBUG::2014-09-25 18:02:09,640::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:09,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:09,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147519 s, 69.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:02:11,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:11,914::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202756 s, 50.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:12,333::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:12,334::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:12,334::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:12,334::__init__::232::IOProcess::(_processLogs) (110254) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:12,337::__init__::232::IOProcess::(_processLogs) (110254) Queuing response >Thread-79::DEBUG::2014-09-25 18:02:12,338::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:02:12,357::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00361135 s, 156 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:12,358::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:12,358::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:12,358::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:12,359::__init__::232::IOProcess::(_processLogs) (110255) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:12,359::__init__::232::IOProcess::(_processLogs) (110255) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:12,361::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:12,361::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:12,361::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:12,362::__init__::232::IOProcess::(_processLogs) (110256) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:12,362::__init__::232::IOProcess::(_processLogs) (110256) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:12,363::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:12,364::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:12,364::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:12,364::__init__::232::IOProcess::(_processLogs) (110257) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:12,365::__init__::232::IOProcess::(_processLogs) (110257) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:12,365::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:12,365::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:12,366::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:12,366::__init__::232::IOProcess::(_processLogs) (110258) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:12,366::__init__::232::IOProcess::(_processLogs) (110258) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:12,660::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:12,662::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71591::DEBUG::2014-09-25 18:02:12,664::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:13,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:13,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174563 s, 58.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:15,686::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:15,687::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71592::DEBUG::2014-09-25 18:02:15,689::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:15,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:16,018::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0217137 s, 47.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:16,322::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:16,324::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:16,326::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 18:02:16,327::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:02:16,328::__init__::232::IOProcess::(_processLogs) (110259) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:16,337::__init__::232::IOProcess::(_processLogs) (110259) Queuing response >Thread-81::DEBUG::2014-09-25 18:02:16,339::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000393336 s, 943 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:16,342::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:16,343::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:16,344::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:16,345::__init__::232::IOProcess::(_processLogs) (110260) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:16,347::__init__::232::IOProcess::(_processLogs) (110260) Queuing response >Dummy-148::DEBUG::2014-09-25 18:02:18,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:18,235::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164385 s, 62.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:18,317::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71593::DEBUG::2014-09-25 18:02:18,319::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:02:18,319::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71593::DEBUG::2014-09-25 18:02:18,321::task::595::Storage.TaskManager.Task::(_updateState) Task=`1dafe782-853c-44d8-9a0a-508aab3926b0`::moving from state init -> state preparing >Thread-71593::INFO::2014-09-25 18:02:18,321::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71593::INFO::2014-09-25 18:02:18,322::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71593::DEBUG::2014-09-25 18:02:18,322::task::1191::Storage.TaskManager.Task::(prepare) Task=`1dafe782-853c-44d8-9a0a-508aab3926b0`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71593::DEBUG::2014-09-25 18:02:18,323::task::595::Storage.TaskManager.Task::(_updateState) Task=`1dafe782-853c-44d8-9a0a-508aab3926b0`::moving from state preparing -> state finished >Thread-71593::DEBUG::2014-09-25 18:02:18,323::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71593::DEBUG::2014-09-25 18:02:18,323::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71593::DEBUG::2014-09-25 18:02:18,323::task::993::Storage.TaskManager.Task::(_decref) Task=`1dafe782-853c-44d8-9a0a-508aab3926b0`::ref 0 aborting False >Thread-71593::DEBUG::2014-09-25 18:02:18,324::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71593::DEBUG::2014-09-25 18:02:18,324::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:18,339::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71594::DEBUG::2014-09-25 18:02:18,340::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:02:18,340::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71594::DEBUG::2014-09-25 18:02:18,343::task::595::Storage.TaskManager.Task::(_updateState) Task=`ee3cb696-6080-4758-9fca-158de78305f8`::moving from state init -> state preparing >Thread-71594::INFO::2014-09-25 18:02:18,343::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71594::DEBUG::2014-09-25 18:02:18,344::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2f62e7c5-2b78-4029-95ff-22d32aba95cf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71594::DEBUG::2014-09-25 18:02:18,344::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71594::DEBUG::2014-09-25 18:02:18,344::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71594::DEBUG::2014-09-25 18:02:18,345::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2f62e7c5-2b78-4029-95ff-22d32aba95cf`::Granted request >Thread-71594::DEBUG::2014-09-25 18:02:18,345::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`ee3cb696-6080-4758-9fca-158de78305f8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71594::DEBUG::2014-09-25 18:02:18,345::task::993::Storage.TaskManager.Task::(_decref) Task=`ee3cb696-6080-4758-9fca-158de78305f8`::ref 1 aborting False >Thread-71594::INFO::2014-09-25 18:02:18,350::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '6831472640', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71594::DEBUG::2014-09-25 18:02:18,350::task::1191::Storage.TaskManager.Task::(prepare) Task=`ee3cb696-6080-4758-9fca-158de78305f8`::finished: {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '6831472640', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71594::DEBUG::2014-09-25 18:02:18,350::task::595::Storage.TaskManager.Task::(_updateState) Task=`ee3cb696-6080-4758-9fca-158de78305f8`::moving from state preparing -> state finished >Thread-71594::DEBUG::2014-09-25 18:02:18,350::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71594::DEBUG::2014-09-25 18:02:18,351::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71594::DEBUG::2014-09-25 18:02:18,351::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71594::DEBUG::2014-09-25 18:02:18,351::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71594::DEBUG::2014-09-25 18:02:18,352::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71594::DEBUG::2014-09-25 18:02:18,352::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71594::DEBUG::2014-09-25 18:02:18,352::task::993::Storage.TaskManager.Task::(_decref) Task=`ee3cb696-6080-4758-9fca-158de78305f8`::ref 0 aborting False >Thread-71594::DEBUG::2014-09-25 18:02:18,353::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'diskfree': '6831472640', 'isoprefix': u'/rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso/fc587b6a-bac3-4668-99de-7f6fd6acce7f/images/11111111-1111-1111-1111-111111111111', 'alerts': [], 'disktotal': '11712593920', 'version': 0}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71594::DEBUG::2014-09-25 18:02:18,353::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:18,705::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:18,706::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71595::DEBUG::2014-09-25 18:02:18,707::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:20,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:20,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020581 s, 49.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:21,721::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:21,723::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71596::DEBUG::2014-09-25 18:02:21,725::task::595::Storage.TaskManager.Task::(_updateState) Task=`abb52666-cf8f-48db-aaca-430faae5e3fe`::moving from state init -> state preparing >Thread-71596::INFO::2014-09-25 18:02:21,726::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71596::INFO::2014-09-25 18:02:21,726::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00361135', 'lastCheck': '9.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449597', 'lastCheck': '63.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000393336', 'lastCheck': '5.4', 'valid': True}} >Thread-71596::DEBUG::2014-09-25 18:02:21,727::task::1191::Storage.TaskManager.Task::(prepare) Task=`abb52666-cf8f-48db-aaca-430faae5e3fe`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00361135', 'lastCheck': '9.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000449597', 'lastCheck': '63.7', 'valid': True}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000393336', 'lastCheck': '5.4', 'valid': True}} >Thread-71596::DEBUG::2014-09-25 18:02:21,727::task::595::Storage.TaskManager.Task::(_updateState) Task=`abb52666-cf8f-48db-aaca-430faae5e3fe`::moving from state preparing -> state finished >Thread-71596::DEBUG::2014-09-25 18:02:21,727::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71596::DEBUG::2014-09-25 18:02:21,727::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71596::DEBUG::2014-09-25 18:02:21,727::task::993::Storage.TaskManager.Task::(_decref) Task=`abb52666-cf8f-48db-aaca-430faae5e3fe`::ref 0 aborting False >Thread-71596::DEBUG::2014-09-25 18:02:21,730::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:21,760::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:21,761::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71597::DEBUG::2014-09-25 18:02:21,763::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:22,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:22,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0246834 s, 41.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:22,369::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:22,370::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:22,371::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:22,372::__init__::232::IOProcess::(_processLogs) (110261) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:22,374::__init__::232::IOProcess::(_processLogs) (110261) Queuing response >Thread-79::DEBUG::2014-09-25 18:02:22,376::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:02:22,394::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00500731 s, 112 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:22,397::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:22,398::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:22,399::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:22,400::__init__::232::IOProcess::(_processLogs) (110262) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:22,402::__init__::232::IOProcess::(_processLogs) (110262) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:22,404::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:22,405::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:22,406::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:22,407::__init__::232::IOProcess::(_processLogs) (110263) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:22,408::__init__::232::IOProcess::(_processLogs) (110263) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:22,410::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:22,411::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:22,412::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:22,413::__init__::232::IOProcess::(_processLogs) (110264) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:22,414::__init__::232::IOProcess::(_processLogs) (110264) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:22,417::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:22,418::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:22,420::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:22,421::__init__::232::IOProcess::(_processLogs) (110265) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:22,422::__init__::232::IOProcess::(_processLogs) (110265) Queuing response >Dummy-148::DEBUG::2014-09-25 18:02:24,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:24,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159137 s, 64.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:25,032::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:25,033::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71598::DEBUG::2014-09-25 18:02:25,035::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:02:26,350::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:26,350::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:26,350::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:26,351::__init__::232::IOProcess::(_processLogs) (110266) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:26,353::__init__::232::IOProcess::(_processLogs) (110266) Queuing response >Thread-81::DEBUG::2014-09-25 18:02:26,354::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:02:26,371::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00227475 s, 163 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:26,372::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:26,372::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:26,372::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:26,373::__init__::232::IOProcess::(_processLogs) (110267) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:26,374::__init__::232::IOProcess::(_processLogs) (110267) Queuing response >Dummy-148::DEBUG::2014-09-25 18:02:26,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:26,514::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0266989 s, 38.4 MB/s\n'; <rc> = 0 >Thread-80::ERROR::2014-09-25 18:02:27,978::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/fileSD.py", line 587, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/outOfProcess.py", line 244, in statvfs > return self._iop.statvfs(path) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 378, in statvfs > resdict = self._sendCommand("statvfs", {"path": path}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::DEBUG::2014-09-25 18:02:27,980::domainMonitor::264::Storage.DomainMonitorThread::(_monitorDomain) Domain fc587b6a-bac3-4668-99de-7f6fd6acce7f changed its status to Invalid >Thread-71599::DEBUG::2014-09-25 18:02:27,981::misc::821::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Emitting event >Thread-71599::DEBUG::2014-09-25 18:02:27,981::misc::831::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Calling registered method `contEIOVms` >Thread-71599::DEBUG::2014-09-25 18:02:27,982::misc::841::Storage.Event.Storage.DomainMonitor.onDomainStateChange::(_emit) Event emitted >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:28,057::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:28,058::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71601::DEBUG::2014-09-25 18:02:28,061::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:28,393::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71602::DEBUG::2014-09-25 18:02:28,394::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:02:28,394::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71602::DEBUG::2014-09-25 18:02:28,396::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ff8edf2-7160-4b67-b590-bae258ee3367`::moving from state init -> state preparing >Thread-71602::INFO::2014-09-25 18:02:28,397::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71602::INFO::2014-09-25 18:02:28,397::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71602::DEBUG::2014-09-25 18:02:28,397::task::1191::Storage.TaskManager.Task::(prepare) Task=`3ff8edf2-7160-4b67-b590-bae258ee3367`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71602::DEBUG::2014-09-25 18:02:28,398::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ff8edf2-7160-4b67-b590-bae258ee3367`::moving from state preparing -> state finished >Thread-71602::DEBUG::2014-09-25 18:02:28,398::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71602::DEBUG::2014-09-25 18:02:28,398::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71602::DEBUG::2014-09-25 18:02:28,398::task::993::Storage.TaskManager.Task::(_decref) Task=`3ff8edf2-7160-4b67-b590-bae258ee3367`::ref 0 aborting False >Thread-71602::DEBUG::2014-09-25 18:02:28,399::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71602::DEBUG::2014-09-25 18:02:28,399::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:28,418::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:28,420::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71603::DEBUG::2014-09-25 18:02:28,420::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71603::DEBUG::2014-09-25 18:02:28,421::task::595::Storage.TaskManager.Task::(_updateState) Task=`90dcd1ea-5ce8-48ee-8387-81ed1c140f74`::moving from state init -> state preparing >Thread-71603::INFO::2014-09-25 18:02:28,422::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71603::DEBUG::2014-09-25 18:02:28,422::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`28e95d2c-0bc3-4818-84c3-8c38625624c0`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71603::DEBUG::2014-09-25 18:02:28,423::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71603::DEBUG::2014-09-25 18:02:28,423::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71603::DEBUG::2014-09-25 18:02:28,423::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`28e95d2c-0bc3-4818-84c3-8c38625624c0`::Granted request >Thread-71603::DEBUG::2014-09-25 18:02:28,424::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`90dcd1ea-5ce8-48ee-8387-81ed1c140f74`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71603::DEBUG::2014-09-25 18:02:28,424::task::993::Storage.TaskManager.Task::(_decref) Task=`90dcd1ea-5ce8-48ee-8387-81ed1c140f74`::ref 1 aborting False >Thread-71603::INFO::2014-09-25 18:02:28,428::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71603::DEBUG::2014-09-25 18:02:28,429::task::1191::Storage.TaskManager.Task::(prepare) Task=`90dcd1ea-5ce8-48ee-8387-81ed1c140f74`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71603::DEBUG::2014-09-25 18:02:28,429::task::595::Storage.TaskManager.Task::(_updateState) Task=`90dcd1ea-5ce8-48ee-8387-81ed1c140f74`::moving from state preparing -> state finished >Thread-71603::DEBUG::2014-09-25 18:02:28,429::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71603::DEBUG::2014-09-25 18:02:28,429::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71603::DEBUG::2014-09-25 18:02:28,430::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71603::DEBUG::2014-09-25 18:02:28,430::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71603::DEBUG::2014-09-25 18:02:28,430::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71603::DEBUG::2014-09-25 18:02:28,430::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71603::DEBUG::2014-09-25 18:02:28,431::task::993::Storage.TaskManager.Task::(_decref) Task=`90dcd1ea-5ce8-48ee-8387-81ed1c140f74`::ref 0 aborting False >Thread-71603::DEBUG::2014-09-25 18:02:28,431::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71603::DEBUG::2014-09-25 18:02:28,432::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:28,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:28,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136174 s, 75.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:02:30,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:30,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017336 s, 59.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:31,136::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:31,137::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71604::DEBUG::2014-09-25 18:02:31,139::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:02:32,420::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:32,421::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:32,421::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:32,421::__init__::232::IOProcess::(_processLogs) (110268) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:32,422::__init__::232::IOProcess::(_processLogs) (110268) Queuing response >Thread-79::DEBUG::2014-09-25 18:02:32,422::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:02:32,433::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00166005 s, 339 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:32,434::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:32,434::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:32,434::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:32,435::__init__::232::IOProcess::(_processLogs) (110269) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:32,438::__init__::232::IOProcess::(_processLogs) (110269) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:32,439::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:32,439::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:32,439::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:32,440::__init__::232::IOProcess::(_processLogs) (110270) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:32,440::__init__::232::IOProcess::(_processLogs) (110270) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:32,441::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:32,441::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:32,441::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:32,442::__init__::232::IOProcess::(_processLogs) (110271) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:32,442::__init__::232::IOProcess::(_processLogs) (110271) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:32,443::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:32,443::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:32,443::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:32,444::__init__::232::IOProcess::(_processLogs) (110272) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:32,444::__init__::232::IOProcess::(_processLogs) (110272) Queuing response >Dummy-148::DEBUG::2014-09-25 18:02:32,624::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:32,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130601 s, 78.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:34,173::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:34,174::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71605::DEBUG::2014-09-25 18:02:34,177::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:34,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:34,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204119 s, 50.2 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 18:02:36,375::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 18:02:36,381::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:36,381::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:36,381::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:36,382::__init__::232::IOProcess::(_processLogs) (110273) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:02:36,382::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:36,382::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:36,383::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:36,383::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:36,383::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:36,385::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:36,387::__init__::232::IOProcess::(_processLogs) (110274) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:02:36,388::__init__::232::IOProcess::(_processLogs) (110275) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:02:36,390::__init__::232::IOProcess::(_processLogs) (110273) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:36,392::__init__::232::IOProcess::(_processLogs) (110275) Queuing response >Thread-81::DEBUG::2014-09-25 18:02:36,395::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 18:02:36,396::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:36,396::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:36,396::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:36,396::__init__::232::IOProcess::(_processLogs) (110276) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 18:02:36,424::__init__::232::IOProcess::(_processLogs) (110276) Queuing response >Thread-81::DEBUG::2014-09-25 18:02:36,424::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:02:36,425::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:36,425::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:36,425::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:36,426::__init__::232::IOProcess::(_processLogs) (110277) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 18:02:36,433::__init__::232::IOProcess::(_processLogs) (110277) Queuing response >Thread-81::DEBUG::2014-09-25 18:02:36,433::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 18:02:36,434::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:36,434::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:36,435::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:36,435::__init__::232::IOProcess::(_processLogs) (110278) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:36,435::__init__::232::IOProcess::(_processLogs) (110278) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:36,436::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:36,436::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:36,436::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:36,437::__init__::232::IOProcess::(_processLogs) (110279) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:02:36,439::__init__::232::IOProcess::(_processLogs) (110279) Queuing response >Thread-81::DEBUG::2014-09-25 18:02:36,439::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 18:02:36,440::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 18:02:36,441::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 18:02:36,441::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:36,442::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:36,442::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:36,442::__init__::232::IOProcess::(_processLogs) (110280) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:36,444::__init__::232::IOProcess::(_processLogs) (110280) Queuing response >Thread-81::DEBUG::2014-09-25 18:02:36,444::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:02:36,458::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00172521 s, 215 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:36,460::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:36,460::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:36,460::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:36,460::__init__::232::IOProcess::(_processLogs) (110281) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:36,463::__init__::232::IOProcess::(_processLogs) (110281) Queuing response >Dummy-148::DEBUG::2014-09-25 18:02:36,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:36,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0266134 s, 38.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:37,195::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:37,197::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71609::DEBUG::2014-09-25 18:02:37,199::task::595::Storage.TaskManager.Task::(_updateState) Task=`c8e53fdd-7710-4b3a-a572-46cf5a848b16`::moving from state init -> state preparing >Thread-71609::INFO::2014-09-25 18:02:37,200::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71609::INFO::2014-09-25 18:02:37,200::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00166005', 'lastCheck': '4.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00172521', 'lastCheck': '0.7', 'valid': True}} >Thread-71609::DEBUG::2014-09-25 18:02:37,201::task::1191::Storage.TaskManager.Task::(prepare) Task=`c8e53fdd-7710-4b3a-a572-46cf5a848b16`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00166005', 'lastCheck': '4.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00172521', 'lastCheck': '0.7', 'valid': True}} >Thread-71609::DEBUG::2014-09-25 18:02:37,201::task::595::Storage.TaskManager.Task::(_updateState) Task=`c8e53fdd-7710-4b3a-a572-46cf5a848b16`::moving from state preparing -> state finished >Thread-71609::DEBUG::2014-09-25 18:02:37,201::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71609::DEBUG::2014-09-25 18:02:37,201::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71609::DEBUG::2014-09-25 18:02:37,201::task::993::Storage.TaskManager.Task::(_decref) Task=`c8e53fdd-7710-4b3a-a572-46cf5a848b16`::ref 0 aborting False >Thread-71609::DEBUG::2014-09-25 18:02:37,204::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:37,241::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:37,243::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71610::DEBUG::2014-09-25 18:02:37,245::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:02:37,982::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:37,982::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:37,983::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:37,983::__init__::232::IOProcess::(_processLogs) (110282) Got request for method 'statvfs' >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:38,502::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71611::DEBUG::2014-09-25 18:02:38,503::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:02:38,503::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71611::DEBUG::2014-09-25 18:02:38,505::task::595::Storage.TaskManager.Task::(_updateState) Task=`3f62db94-9366-4382-8451-1c27499fb594`::moving from state init -> state preparing >Thread-71611::INFO::2014-09-25 18:02:38,505::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71611::INFO::2014-09-25 18:02:38,506::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71611::DEBUG::2014-09-25 18:02:38,506::task::1191::Storage.TaskManager.Task::(prepare) Task=`3f62db94-9366-4382-8451-1c27499fb594`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71611::DEBUG::2014-09-25 18:02:38,506::task::595::Storage.TaskManager.Task::(_updateState) Task=`3f62db94-9366-4382-8451-1c27499fb594`::moving from state preparing -> state finished >Thread-71611::DEBUG::2014-09-25 18:02:38,506::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71611::DEBUG::2014-09-25 18:02:38,507::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71611::DEBUG::2014-09-25 18:02:38,507::task::993::Storage.TaskManager.Task::(_decref) Task=`3f62db94-9366-4382-8451-1c27499fb594`::ref 0 aborting False >Thread-71611::DEBUG::2014-09-25 18:02:38,507::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71611::DEBUG::2014-09-25 18:02:38,507::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:38,580::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71612::DEBUG::2014-09-25 18:02:38,581::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:02:38,582::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71612::DEBUG::2014-09-25 18:02:38,584::task::595::Storage.TaskManager.Task::(_updateState) Task=`75cdcbc9-5f51-4377-9d01-456901416322`::moving from state init -> state preparing >Thread-71612::INFO::2014-09-25 18:02:38,584::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71612::DEBUG::2014-09-25 18:02:38,585::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cbbaeff2-6778-4f3b-b715-03192ee1d47b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71612::DEBUG::2014-09-25 18:02:38,585::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71612::DEBUG::2014-09-25 18:02:38,585::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71612::DEBUG::2014-09-25 18:02:38,586::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cbbaeff2-6778-4f3b-b715-03192ee1d47b`::Granted request >Thread-71612::DEBUG::2014-09-25 18:02:38,586::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`75cdcbc9-5f51-4377-9d01-456901416322`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71612::DEBUG::2014-09-25 18:02:38,586::task::993::Storage.TaskManager.Task::(_decref) Task=`75cdcbc9-5f51-4377-9d01-456901416322`::ref 1 aborting False >Thread-71612::INFO::2014-09-25 18:02:38,591::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71612::DEBUG::2014-09-25 18:02:38,591::task::1191::Storage.TaskManager.Task::(prepare) Task=`75cdcbc9-5f51-4377-9d01-456901416322`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71612::DEBUG::2014-09-25 18:02:38,591::task::595::Storage.TaskManager.Task::(_updateState) Task=`75cdcbc9-5f51-4377-9d01-456901416322`::moving from state preparing -> state finished >Thread-71612::DEBUG::2014-09-25 18:02:38,592::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71612::DEBUG::2014-09-25 18:02:38,592::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71612::DEBUG::2014-09-25 18:02:38,592::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71612::DEBUG::2014-09-25 18:02:38,592::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71612::DEBUG::2014-09-25 18:02:38,593::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71612::DEBUG::2014-09-25 18:02:38,593::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71612::DEBUG::2014-09-25 18:02:38,593::task::993::Storage.TaskManager.Task::(_decref) Task=`75cdcbc9-5f51-4377-9d01-456901416322`::ref 0 aborting False >Thread-71612::DEBUG::2014-09-25 18:02:38,594::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71612::DEBUG::2014-09-25 18:02:38,594::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:38,771::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:38,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.027423 s, 37.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:40,361::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:40,363::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71613::DEBUG::2014-09-25 18:02:40,365::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:40,828::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:40,863::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218563 s, 46.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:42,446::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:42,446::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:42,447::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:42,447::__init__::232::IOProcess::(_processLogs) (110283) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:42,448::__init__::232::IOProcess::(_processLogs) (110283) Queuing response >Thread-79::DEBUG::2014-09-25 18:02:42,449::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:02:42,469::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00260458 s, 216 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:42,470::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:42,470::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:42,470::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:42,471::__init__::232::IOProcess::(_processLogs) (110284) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:42,474::__init__::232::IOProcess::(_processLogs) (110284) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:42,475::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:42,475::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:42,476::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:42,476::__init__::232::IOProcess::(_processLogs) (110285) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:42,477::__init__::232::IOProcess::(_processLogs) (110285) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:42,477::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:42,477::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:42,478::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:42,479::__init__::232::IOProcess::(_processLogs) (110286) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:42,479::__init__::232::IOProcess::(_processLogs) (110286) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:42,479::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:42,479::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:42,480::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:42,481::__init__::232::IOProcess::(_processLogs) (110287) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:42,481::__init__::232::IOProcess::(_processLogs) (110287) Queuing response >Dummy-148::DEBUG::2014-09-25 18:02:42,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:42,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142878 s, 71.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:43,382::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:43,383::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71614::DEBUG::2014-09-25 18:02:43,385::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:44,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:44,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0127839 s, 80.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:46,404::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:46,407::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71615::DEBUG::2014-09-25 18:02:46,413::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:02:46,466::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:46,467::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:46,468::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:46,469::__init__::232::IOProcess::(_processLogs) (110288) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:46,471::__init__::232::IOProcess::(_processLogs) (110288) Queuing response >Thread-81::DEBUG::2014-09-25 18:02:46,473::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:02:46,494::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00513382 s, 72.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:46,497::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:46,498::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:46,499::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:46,500::__init__::232::IOProcess::(_processLogs) (110289) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:46,502::__init__::232::IOProcess::(_processLogs) (110289) Queuing response >Dummy-148::DEBUG::2014-09-25 18:02:46,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:47,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0268287 s, 38.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:48,643::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71616::DEBUG::2014-09-25 18:02:48,645::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:02:48,645::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71616::DEBUG::2014-09-25 18:02:48,647::task::595::Storage.TaskManager.Task::(_updateState) Task=`5c258e04-f143-46da-819e-6e886eb31add`::moving from state init -> state preparing >Thread-71616::INFO::2014-09-25 18:02:48,648::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71616::INFO::2014-09-25 18:02:48,648::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71616::DEBUG::2014-09-25 18:02:48,648::task::1191::Storage.TaskManager.Task::(prepare) Task=`5c258e04-f143-46da-819e-6e886eb31add`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71616::DEBUG::2014-09-25 18:02:48,649::task::595::Storage.TaskManager.Task::(_updateState) Task=`5c258e04-f143-46da-819e-6e886eb31add`::moving from state preparing -> state finished >Thread-71616::DEBUG::2014-09-25 18:02:48,649::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71616::DEBUG::2014-09-25 18:02:48,649::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71616::DEBUG::2014-09-25 18:02:48,649::task::993::Storage.TaskManager.Task::(_decref) Task=`5c258e04-f143-46da-819e-6e886eb31add`::ref 0 aborting False >Thread-71616::DEBUG::2014-09-25 18:02:48,650::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71616::DEBUG::2014-09-25 18:02:48,650::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:48,719::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71617::DEBUG::2014-09-25 18:02:48,720::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:02:48,720::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71617::DEBUG::2014-09-25 18:02:48,722::task::595::Storage.TaskManager.Task::(_updateState) Task=`76dc55c5-7a01-4d37-96ea-b11357c7a3b9`::moving from state init -> state preparing >Thread-71617::INFO::2014-09-25 18:02:48,722::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71617::DEBUG::2014-09-25 18:02:48,723::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6ebd8592-0678-4ca9-acd3-a04560a0069c`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71617::DEBUG::2014-09-25 18:02:48,723::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71617::DEBUG::2014-09-25 18:02:48,723::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71617::DEBUG::2014-09-25 18:02:48,724::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6ebd8592-0678-4ca9-acd3-a04560a0069c`::Granted request >Thread-71617::DEBUG::2014-09-25 18:02:48,724::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`76dc55c5-7a01-4d37-96ea-b11357c7a3b9`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71617::DEBUG::2014-09-25 18:02:48,725::task::993::Storage.TaskManager.Task::(_decref) Task=`76dc55c5-7a01-4d37-96ea-b11357c7a3b9`::ref 1 aborting False >Thread-71617::INFO::2014-09-25 18:02:48,729::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71617::DEBUG::2014-09-25 18:02:48,729::task::1191::Storage.TaskManager.Task::(prepare) Task=`76dc55c5-7a01-4d37-96ea-b11357c7a3b9`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71617::DEBUG::2014-09-25 18:02:48,729::task::595::Storage.TaskManager.Task::(_updateState) Task=`76dc55c5-7a01-4d37-96ea-b11357c7a3b9`::moving from state preparing -> state finished >Thread-71617::DEBUG::2014-09-25 18:02:48,730::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71617::DEBUG::2014-09-25 18:02:48,730::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71617::DEBUG::2014-09-25 18:02:48,730::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71617::DEBUG::2014-09-25 18:02:48,730::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71617::DEBUG::2014-09-25 18:02:48,731::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71617::DEBUG::2014-09-25 18:02:48,731::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71617::DEBUG::2014-09-25 18:02:48,731::task::993::Storage.TaskManager.Task::(_decref) Task=`76dc55c5-7a01-4d37-96ea-b11357c7a3b9`::ref 0 aborting False >Thread-71617::DEBUG::2014-09-25 18:02:48,732::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71617::DEBUG::2014-09-25 18:02:48,732::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:49,035::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:49,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132571 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:49,436::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:49,438::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71618::DEBUG::2014-09-25 18:02:49,440::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:51,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:51,111::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203193 s, 50.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:52,454::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:52,456::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71619::DEBUG::2014-09-25 18:02:52,458::task::595::Storage.TaskManager.Task::(_updateState) Task=`29b8b17e-fcf7-4b2d-940e-c3c271ca91da`::moving from state init -> state preparing >Thread-71619::INFO::2014-09-25 18:02:52,459::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71619::INFO::2014-09-25 18:02:52,459::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00260458', 'lastCheck': '10.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '24.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00513382', 'lastCheck': '6.0', 'valid': True}} >Thread-71619::DEBUG::2014-09-25 18:02:52,459::task::1191::Storage.TaskManager.Task::(prepare) Task=`29b8b17e-fcf7-4b2d-940e-c3c271ca91da`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00260458', 'lastCheck': '10.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '24.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00513382', 'lastCheck': '6.0', 'valid': True}} >Thread-71619::DEBUG::2014-09-25 18:02:52,460::task::595::Storage.TaskManager.Task::(_updateState) Task=`29b8b17e-fcf7-4b2d-940e-c3c271ca91da`::moving from state preparing -> state finished >Thread-71619::DEBUG::2014-09-25 18:02:52,460::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71619::DEBUG::2014-09-25 18:02:52,460::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71619::DEBUG::2014-09-25 18:02:52,460::task::993::Storage.TaskManager.Task::(_decref) Task=`29b8b17e-fcf7-4b2d-940e-c3c271ca91da`::ref 0 aborting False >Thread-71619::DEBUG::2014-09-25 18:02:52,463::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:52,480::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:52,481::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-29::DEBUG::2014-09-25 18:02:52,484::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:52,485::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:52,486::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-71620::DEBUG::2014-09-25 18:02:52,487::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:02:52,488::__init__::232::IOProcess::(_processLogs) (110290) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:52,490::__init__::232::IOProcess::(_processLogs) (110290) Queuing response >Thread-79::DEBUG::2014-09-25 18:02:52,492::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:02:52,500::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000402489 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:52,501::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:52,502::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:52,502::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:52,504::__init__::232::IOProcess::(_processLogs) (110291) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:52,504::__init__::232::IOProcess::(_processLogs) (110291) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:52,504::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:52,505::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:52,505::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:52,506::__init__::232::IOProcess::(_processLogs) (110292) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:52,506::__init__::232::IOProcess::(_processLogs) (110292) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:52,506::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:52,507::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:52,507::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:52,508::__init__::232::IOProcess::(_processLogs) (110293) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:52,508::__init__::232::IOProcess::(_processLogs) (110293) Queuing response >Thread-29::DEBUG::2014-09-25 18:02:52,508::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:52,509::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:52,509::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:52,510::__init__::232::IOProcess::(_processLogs) (110294) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:02:52,510::__init__::232::IOProcess::(_processLogs) (110294) Queuing response >Dummy-148::DEBUG::2014-09-25 18:02:53,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:53,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194213 s, 52.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:02:55,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:55,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141308 s, 72.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:55,534::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:55,536::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71621::DEBUG::2014-09-25 18:02:55,538::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:02:56,505::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:56,505::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:56,506::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:56,506::__init__::232::IOProcess::(_processLogs) (110295) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:56,506::__init__::232::IOProcess::(_processLogs) (110295) Queuing response >Thread-81::DEBUG::2014-09-25 18:02:56,507::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:02:56,516::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000442901 s, 838 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:02:56,518::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:02:56,518::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:02:56,518::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:02:56,520::__init__::232::IOProcess::(_processLogs) (110296) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:02:56,520::__init__::232::IOProcess::(_processLogs) (110296) Queuing response >Dummy-148::DEBUG::2014-09-25 18:02:57,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:57,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0305537 s, 33.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:58,552::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:02:58,553::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71622::DEBUG::2014-09-25 18:02:58,555::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:58,757::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71623::DEBUG::2014-09-25 18:02:58,758::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:02:58,758::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71623::DEBUG::2014-09-25 18:02:58,760::task::595::Storage.TaskManager.Task::(_updateState) Task=`0825f3b0-3764-46ab-84a4-2a181e463741`::moving from state init -> state preparing >Thread-71623::INFO::2014-09-25 18:02:58,760::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71623::INFO::2014-09-25 18:02:58,761::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71623::DEBUG::2014-09-25 18:02:58,761::task::1191::Storage.TaskManager.Task::(prepare) Task=`0825f3b0-3764-46ab-84a4-2a181e463741`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71623::DEBUG::2014-09-25 18:02:58,761::task::595::Storage.TaskManager.Task::(_updateState) Task=`0825f3b0-3764-46ab-84a4-2a181e463741`::moving from state preparing -> state finished >Thread-71623::DEBUG::2014-09-25 18:02:58,762::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71623::DEBUG::2014-09-25 18:02:58,762::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71623::DEBUG::2014-09-25 18:02:58,762::task::993::Storage.TaskManager.Task::(_decref) Task=`0825f3b0-3764-46ab-84a4-2a181e463741`::ref 0 aborting False >Thread-71623::DEBUG::2014-09-25 18:02:58,762::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71623::DEBUG::2014-09-25 18:02:58,763::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:02:58,774::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71624::DEBUG::2014-09-25 18:02:58,775::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:02:58,775::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71624::DEBUG::2014-09-25 18:02:58,777::task::595::Storage.TaskManager.Task::(_updateState) Task=`1d45e83f-9b21-4950-941a-fcfd48eaa27d`::moving from state init -> state preparing >Thread-71624::INFO::2014-09-25 18:02:58,777::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71624::DEBUG::2014-09-25 18:02:58,778::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`574ec4e6-e5ad-4e75-8fe2-d7773b71bba7`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71624::DEBUG::2014-09-25 18:02:58,778::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71624::DEBUG::2014-09-25 18:02:58,779::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71624::DEBUG::2014-09-25 18:02:58,779::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`574ec4e6-e5ad-4e75-8fe2-d7773b71bba7`::Granted request >Thread-71624::DEBUG::2014-09-25 18:02:58,780::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`1d45e83f-9b21-4950-941a-fcfd48eaa27d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71624::DEBUG::2014-09-25 18:02:58,780::task::993::Storage.TaskManager.Task::(_decref) Task=`1d45e83f-9b21-4950-941a-fcfd48eaa27d`::ref 1 aborting False >Thread-71624::INFO::2014-09-25 18:02:58,784::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71624::DEBUG::2014-09-25 18:02:58,784::task::1191::Storage.TaskManager.Task::(prepare) Task=`1d45e83f-9b21-4950-941a-fcfd48eaa27d`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71624::DEBUG::2014-09-25 18:02:58,785::task::595::Storage.TaskManager.Task::(_updateState) Task=`1d45e83f-9b21-4950-941a-fcfd48eaa27d`::moving from state preparing -> state finished >Thread-71624::DEBUG::2014-09-25 18:02:58,785::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71624::DEBUG::2014-09-25 18:02:58,785::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71624::DEBUG::2014-09-25 18:02:58,785::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71624::DEBUG::2014-09-25 18:02:58,786::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71624::DEBUG::2014-09-25 18:02:58,786::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71624::DEBUG::2014-09-25 18:02:58,786::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71624::DEBUG::2014-09-25 18:02:58,787::task::993::Storage.TaskManager.Task::(_decref) Task=`1d45e83f-9b21-4950-941a-fcfd48eaa27d`::ref 0 aborting False >Thread-71624::DEBUG::2014-09-25 18:02:58,787::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71624::DEBUG::2014-09-25 18:02:58,787::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:02:59,288::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:02:59,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194682 s, 52.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:03:01,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:01,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165671 s, 61.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:01,574::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:01,576::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71625::DEBUG::2014-09-25 18:03:01,578::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:03:02,511::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:02,511::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:02,512::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:02,512::__init__::232::IOProcess::(_processLogs) (110297) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:02,514::__init__::232::IOProcess::(_processLogs) (110297) Queuing response >Thread-79::DEBUG::2014-09-25 18:03:02,515::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:03:02,534::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00573598 s, 98.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:02,535::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:02,535::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:02,535::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:02,536::__init__::232::IOProcess::(_processLogs) (110298) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:02,538::__init__::232::IOProcess::(_processLogs) (110298) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:02,539::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:02,540::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:02,540::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:02,541::__init__::232::IOProcess::(_processLogs) (110299) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:02,541::__init__::232::IOProcess::(_processLogs) (110299) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:02,541::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:02,542::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:02,542::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:02,543::__init__::232::IOProcess::(_processLogs) (110300) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:02,543::__init__::232::IOProcess::(_processLogs) (110300) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:02,543::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:02,544::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:02,545::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:02,545::__init__::232::IOProcess::(_processLogs) (110301) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:02,545::__init__::232::IOProcess::(_processLogs) (110301) Queuing response >Dummy-148::DEBUG::2014-09-25 18:03:03,390::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:03,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0245933 s, 41.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:04,593::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:04,595::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71626::DEBUG::2014-09-25 18:03:04,597::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:05,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:05,479::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213371 s, 48.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:06,521::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:06,521::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:06,521::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:06,521::__init__::232::IOProcess::(_processLogs) (110302) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:06,528::__init__::232::IOProcess::(_processLogs) (110302) Queuing response >Thread-81::DEBUG::2014-09-25 18:03:06,529::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:03:06,556::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00395196 s, 93.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:06,557::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:06,557::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:06,557::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:06,558::__init__::232::IOProcess::(_processLogs) (110303) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:06,562::__init__::232::IOProcess::(_processLogs) (110303) Queuing response >Dummy-148::DEBUG::2014-09-25 18:03:07,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:07,529::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184157 s, 55.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:07,609::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:07,611::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71627::DEBUG::2014-09-25 18:03:07,613::task::595::Storage.TaskManager.Task::(_updateState) Task=`b71f1011-a2bd-42f3-bc8b-88e4a00e6751`::moving from state init -> state preparing >Thread-71627::INFO::2014-09-25 18:03:07,614::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71627::INFO::2014-09-25 18:03:07,614::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00573598', 'lastCheck': '5.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '39.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00395196', 'lastCheck': '1.1', 'valid': True}} >Thread-71627::DEBUG::2014-09-25 18:03:07,614::task::1191::Storage.TaskManager.Task::(prepare) Task=`b71f1011-a2bd-42f3-bc8b-88e4a00e6751`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00573598', 'lastCheck': '5.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '39.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00395196', 'lastCheck': '1.1', 'valid': True}} >Thread-71627::DEBUG::2014-09-25 18:03:07,615::task::595::Storage.TaskManager.Task::(_updateState) Task=`b71f1011-a2bd-42f3-bc8b-88e4a00e6751`::moving from state preparing -> state finished >Thread-71627::DEBUG::2014-09-25 18:03:07,615::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71627::DEBUG::2014-09-25 18:03:07,615::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71627::DEBUG::2014-09-25 18:03:07,615::task::993::Storage.TaskManager.Task::(_decref) Task=`b71f1011-a2bd-42f3-bc8b-88e4a00e6751`::ref 0 aborting False >Thread-71627::DEBUG::2014-09-25 18:03:07,618::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:07,646::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:07,647::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71628::DEBUG::2014-09-25 18:03:07,649::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:08,812::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:08,813::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71629::DEBUG::2014-09-25 18:03:08,813::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71629::DEBUG::2014-09-25 18:03:08,814::task::595::Storage.TaskManager.Task::(_updateState) Task=`c89b5c1b-0d41-4f34-9787-2cf88bf9217f`::moving from state init -> state preparing >Thread-71629::INFO::2014-09-25 18:03:08,815::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71629::INFO::2014-09-25 18:03:08,815::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71629::DEBUG::2014-09-25 18:03:08,816::task::1191::Storage.TaskManager.Task::(prepare) Task=`c89b5c1b-0d41-4f34-9787-2cf88bf9217f`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71629::DEBUG::2014-09-25 18:03:08,816::task::595::Storage.TaskManager.Task::(_updateState) Task=`c89b5c1b-0d41-4f34-9787-2cf88bf9217f`::moving from state preparing -> state finished >Thread-71629::DEBUG::2014-09-25 18:03:08,816::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71629::DEBUG::2014-09-25 18:03:08,816::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71629::DEBUG::2014-09-25 18:03:08,816::task::993::Storage.TaskManager.Task::(_decref) Task=`c89b5c1b-0d41-4f34-9787-2cf88bf9217f`::ref 0 aborting False >Thread-71629::DEBUG::2014-09-25 18:03:08,817::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71629::DEBUG::2014-09-25 18:03:08,817::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:08,850::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:08,851::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71630::DEBUG::2014-09-25 18:03:08,851::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71630::DEBUG::2014-09-25 18:03:08,853::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a04cd3e-c925-4c91-a0d3-deb6d24f7ef0`::moving from state init -> state preparing >Thread-71630::INFO::2014-09-25 18:03:08,853::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71630::DEBUG::2014-09-25 18:03:08,854::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`430b855c-9f12-41c0-9cd0-22e9f65166c9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71630::DEBUG::2014-09-25 18:03:08,854::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71630::DEBUG::2014-09-25 18:03:08,854::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71630::DEBUG::2014-09-25 18:03:08,855::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`430b855c-9f12-41c0-9cd0-22e9f65166c9`::Granted request >Thread-71630::DEBUG::2014-09-25 18:03:08,855::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`4a04cd3e-c925-4c91-a0d3-deb6d24f7ef0`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71630::DEBUG::2014-09-25 18:03:08,855::task::993::Storage.TaskManager.Task::(_decref) Task=`4a04cd3e-c925-4c91-a0d3-deb6d24f7ef0`::ref 1 aborting False >Thread-71630::INFO::2014-09-25 18:03:08,860::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71630::DEBUG::2014-09-25 18:03:08,860::task::1191::Storage.TaskManager.Task::(prepare) Task=`4a04cd3e-c925-4c91-a0d3-deb6d24f7ef0`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71630::DEBUG::2014-09-25 18:03:08,860::task::595::Storage.TaskManager.Task::(_updateState) Task=`4a04cd3e-c925-4c91-a0d3-deb6d24f7ef0`::moving from state preparing -> state finished >Thread-71630::DEBUG::2014-09-25 18:03:08,861::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71630::DEBUG::2014-09-25 18:03:08,861::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71630::DEBUG::2014-09-25 18:03:08,861::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71630::DEBUG::2014-09-25 18:03:08,861::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71630::DEBUG::2014-09-25 18:03:08,862::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71630::DEBUG::2014-09-25 18:03:08,862::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71630::DEBUG::2014-09-25 18:03:08,862::task::993::Storage.TaskManager.Task::(_decref) Task=`4a04cd3e-c925-4c91-a0d3-deb6d24f7ef0`::ref 0 aborting False >Thread-71630::DEBUG::2014-09-25 18:03:08,863::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71630::DEBUG::2014-09-25 18:03:08,863::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:09,549::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:09,574::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159533 s, 64.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:10,705::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:10,707::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71631::DEBUG::2014-09-25 18:03:10,709::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:11,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:11,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132972 s, 77.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:12,546::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:12,546::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:12,546::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:12,547::__init__::232::IOProcess::(_processLogs) (110304) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:12,547::__init__::232::IOProcess::(_processLogs) (110304) Queuing response >Thread-79::DEBUG::2014-09-25 18:03:12,548::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:03:12,557::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000431402 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:12,559::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:12,559::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:12,559::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:12,559::__init__::232::IOProcess::(_processLogs) (110305) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:12,560::__init__::232::IOProcess::(_processLogs) (110305) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:12,561::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:12,562::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:12,562::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:12,562::__init__::232::IOProcess::(_processLogs) (110306) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:12,562::__init__::232::IOProcess::(_processLogs) (110306) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:12,563::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:12,564::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:12,564::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:12,564::__init__::232::IOProcess::(_processLogs) (110307) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:12,565::__init__::232::IOProcess::(_processLogs) (110307) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:12,565::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:12,565::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:12,566::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:12,566::__init__::232::IOProcess::(_processLogs) (110308) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:12,566::__init__::232::IOProcess::(_processLogs) (110308) Queuing response >Dummy-148::DEBUG::2014-09-25 18:03:13,636::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:13,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0190422 s, 53.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:13,735::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:13,736::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71632::DEBUG::2014-09-25 18:03:13,739::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:15,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:15,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015361 s, 66.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:16,565::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:16,566::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-81::DEBUG::2014-09-25 18:03:16,567::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:03:16,568::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:16,575::__init__::232::IOProcess::(_processLogs) (110309) Got request for method 'statvfs' >Thread-81::DEBUG::2014-09-25 18:03:16,576::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000334649 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:16,577::__init__::232::IOProcess::(_processLogs) (110309) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:16,579::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:16,580::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:16,581::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:16,581::__init__::232::IOProcess::(_processLogs) (110310) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:16,583::__init__::232::IOProcess::(_processLogs) (110310) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:16,754::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:16,756::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71633::DEBUG::2014-09-25 18:03:16,761::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:17,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:17,773::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183613 s, 55.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:18,897::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71634::DEBUG::2014-09-25 18:03:18,898::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:03:18,898::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71634::DEBUG::2014-09-25 18:03:18,902::task::595::Storage.TaskManager.Task::(_updateState) Task=`ffdfb46d-6ab4-47c9-b043-323f2d3bc910`::moving from state init -> state preparing >Thread-71634::INFO::2014-09-25 18:03:18,902::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71634::INFO::2014-09-25 18:03:18,903::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71634::DEBUG::2014-09-25 18:03:18,903::task::1191::Storage.TaskManager.Task::(prepare) Task=`ffdfb46d-6ab4-47c9-b043-323f2d3bc910`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71634::DEBUG::2014-09-25 18:03:18,903::task::595::Storage.TaskManager.Task::(_updateState) Task=`ffdfb46d-6ab4-47c9-b043-323f2d3bc910`::moving from state preparing -> state finished >Thread-71634::DEBUG::2014-09-25 18:03:18,903::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71634::DEBUG::2014-09-25 18:03:18,903::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71634::DEBUG::2014-09-25 18:03:18,904::task::993::Storage.TaskManager.Task::(_decref) Task=`ffdfb46d-6ab4-47c9-b043-323f2d3bc910`::ref 0 aborting False >Thread-71634::DEBUG::2014-09-25 18:03:18,904::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71634::DEBUG::2014-09-25 18:03:18,904::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:19,005::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71635::DEBUG::2014-09-25 18:03:19,006::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:03:19,006::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71635::DEBUG::2014-09-25 18:03:19,008::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c582367-37c7-4fdf-a0ad-b0fc5effeb0a`::moving from state init -> state preparing >Thread-71635::INFO::2014-09-25 18:03:19,008::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71635::DEBUG::2014-09-25 18:03:19,009::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`642e6a61-fe8f-4828-9db4-fa1025121719`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71635::DEBUG::2014-09-25 18:03:19,009::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71635::DEBUG::2014-09-25 18:03:19,010::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71635::DEBUG::2014-09-25 18:03:19,010::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`642e6a61-fe8f-4828-9db4-fa1025121719`::Granted request >Thread-71635::DEBUG::2014-09-25 18:03:19,011::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3c582367-37c7-4fdf-a0ad-b0fc5effeb0a`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71635::DEBUG::2014-09-25 18:03:19,011::task::993::Storage.TaskManager.Task::(_decref) Task=`3c582367-37c7-4fdf-a0ad-b0fc5effeb0a`::ref 1 aborting False >Thread-71635::INFO::2014-09-25 18:03:19,015::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71635::DEBUG::2014-09-25 18:03:19,015::task::1191::Storage.TaskManager.Task::(prepare) Task=`3c582367-37c7-4fdf-a0ad-b0fc5effeb0a`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71635::DEBUG::2014-09-25 18:03:19,016::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c582367-37c7-4fdf-a0ad-b0fc5effeb0a`::moving from state preparing -> state finished >Thread-71635::DEBUG::2014-09-25 18:03:19,016::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71635::DEBUG::2014-09-25 18:03:19,016::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71635::DEBUG::2014-09-25 18:03:19,016::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71635::DEBUG::2014-09-25 18:03:19,017::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71635::DEBUG::2014-09-25 18:03:19,017::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71635::DEBUG::2014-09-25 18:03:19,017::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71635::DEBUG::2014-09-25 18:03:19,018::task::993::Storage.TaskManager.Task::(_decref) Task=`3c582367-37c7-4fdf-a0ad-b0fc5effeb0a`::ref 0 aborting False >Thread-71635::DEBUG::2014-09-25 18:03:19,018::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71635::DEBUG::2014-09-25 18:03:19,019::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:19,778::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:19,780::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71636::DEBUG::2014-09-25 18:03:19,781::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:19,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:19,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016252 s, 63.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:03:21,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:21,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189079 s, 54.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:22,568::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:22,568::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:22,568::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:22,569::__init__::232::IOProcess::(_processLogs) (110311) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:22,579::__init__::232::IOProcess::(_processLogs) (110311) Queuing response >Thread-79::DEBUG::2014-09-25 18:03:22,580::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:03:22,607::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00716823 s, 78.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:22,608::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:22,609::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:22,609::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:22,609::__init__::232::IOProcess::(_processLogs) (110312) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:22,615::__init__::232::IOProcess::(_processLogs) (110312) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:22,616::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:22,617::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:22,617::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:22,618::__init__::232::IOProcess::(_processLogs) (110313) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:22,618::__init__::232::IOProcess::(_processLogs) (110313) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:22,618::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:22,619::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:22,619::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:22,620::__init__::232::IOProcess::(_processLogs) (110314) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:22,620::__init__::232::IOProcess::(_processLogs) (110314) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:22,620::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:22,621::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:22,622::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:22,622::__init__::232::IOProcess::(_processLogs) (110315) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:22,622::__init__::232::IOProcess::(_processLogs) (110315) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:22,796::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:22,797::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71637::DEBUG::2014-09-25 18:03:22,800::task::595::Storage.TaskManager.Task::(_updateState) Task=`09069613-329c-4668-8450-d32dba03033a`::moving from state init -> state preparing >Thread-71637::INFO::2014-09-25 18:03:22,800::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71637::INFO::2014-09-25 18:03:22,801::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00716823', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '54.8', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000334649', 'lastCheck': '6.2', 'valid': True}} >Thread-71637::DEBUG::2014-09-25 18:03:22,801::task::1191::Storage.TaskManager.Task::(prepare) Task=`09069613-329c-4668-8450-d32dba03033a`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00716823', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '54.8', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000334649', 'lastCheck': '6.2', 'valid': True}} >Thread-71637::DEBUG::2014-09-25 18:03:22,801::task::595::Storage.TaskManager.Task::(_updateState) Task=`09069613-329c-4668-8450-d32dba03033a`::moving from state preparing -> state finished >Thread-71637::DEBUG::2014-09-25 18:03:22,801::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71637::DEBUG::2014-09-25 18:03:22,801::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71637::DEBUG::2014-09-25 18:03:22,802::task::993::Storage.TaskManager.Task::(_decref) Task=`09069613-329c-4668-8450-d32dba03033a`::ref 0 aborting False >Thread-71637::DEBUG::2014-09-25 18:03:22,804::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:22,821::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:22,822::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71638::DEBUG::2014-09-25 18:03:22,824::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:23,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:23,930::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156703 s, 65.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:25,891::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:25,892::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71639::DEBUG::2014-09-25 18:03:25,894::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:25,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:25,982::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162048 s, 63.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:26,585::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:26,585::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:26,586::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:26,586::__init__::232::IOProcess::(_processLogs) (110316) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:26,586::__init__::232::IOProcess::(_processLogs) (110316) Queuing response >Thread-81::DEBUG::2014-09-25 18:03:26,588::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:03:26,606::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00717219 s, 51.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:26,607::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:26,608::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:26,608::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:26,608::__init__::232::IOProcess::(_processLogs) (110317) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:26,615::__init__::232::IOProcess::(_processLogs) (110317) Queuing response >Dummy-148::DEBUG::2014-09-25 18:03:28,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:28,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134346 s, 76.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:28,909::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:28,911::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71640::DEBUG::2014-09-25 18:03:28,913::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:29,049::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:29,052::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71641::DEBUG::2014-09-25 18:03:29,053::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71641::DEBUG::2014-09-25 18:03:29,060::task::595::Storage.TaskManager.Task::(_updateState) Task=`efea9e2c-1da2-4c1b-8ac2-acf52d5d9425`::moving from state init -> state preparing >Thread-71641::INFO::2014-09-25 18:03:29,061::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71641::INFO::2014-09-25 18:03:29,064::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71641::DEBUG::2014-09-25 18:03:29,065::task::1191::Storage.TaskManager.Task::(prepare) Task=`efea9e2c-1da2-4c1b-8ac2-acf52d5d9425`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71641::DEBUG::2014-09-25 18:03:29,066::task::595::Storage.TaskManager.Task::(_updateState) Task=`efea9e2c-1da2-4c1b-8ac2-acf52d5d9425`::moving from state preparing -> state finished >Thread-71641::DEBUG::2014-09-25 18:03:29,067::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71641::DEBUG::2014-09-25 18:03:29,068::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71641::DEBUG::2014-09-25 18:03:29,069::task::993::Storage.TaskManager.Task::(_decref) Task=`efea9e2c-1da2-4c1b-8ac2-acf52d5d9425`::ref 0 aborting False >Thread-71641::DEBUG::2014-09-25 18:03:29,071::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71641::DEBUG::2014-09-25 18:03:29,072::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:29,085::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71642::DEBUG::2014-09-25 18:03:29,086::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:03:29,086::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71642::DEBUG::2014-09-25 18:03:29,088::task::595::Storage.TaskManager.Task::(_updateState) Task=`04bd2f5e-5d5c-4d58-b5a5-7c080a8a8ab4`::moving from state init -> state preparing >Thread-71642::INFO::2014-09-25 18:03:29,088::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71642::DEBUG::2014-09-25 18:03:29,089::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2ae77245-c795-4ee7-9c73-5b8300775b65`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71642::DEBUG::2014-09-25 18:03:29,089::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71642::DEBUG::2014-09-25 18:03:29,090::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71642::DEBUG::2014-09-25 18:03:29,090::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2ae77245-c795-4ee7-9c73-5b8300775b65`::Granted request >Thread-71642::DEBUG::2014-09-25 18:03:29,091::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`04bd2f5e-5d5c-4d58-b5a5-7c080a8a8ab4`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71642::DEBUG::2014-09-25 18:03:29,091::task::993::Storage.TaskManager.Task::(_decref) Task=`04bd2f5e-5d5c-4d58-b5a5-7c080a8a8ab4`::ref 1 aborting False >Thread-71642::INFO::2014-09-25 18:03:29,095::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71642::DEBUG::2014-09-25 18:03:29,095::task::1191::Storage.TaskManager.Task::(prepare) Task=`04bd2f5e-5d5c-4d58-b5a5-7c080a8a8ab4`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71642::DEBUG::2014-09-25 18:03:29,096::task::595::Storage.TaskManager.Task::(_updateState) Task=`04bd2f5e-5d5c-4d58-b5a5-7c080a8a8ab4`::moving from state preparing -> state finished >Thread-71642::DEBUG::2014-09-25 18:03:29,096::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71642::DEBUG::2014-09-25 18:03:29,096::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71642::DEBUG::2014-09-25 18:03:29,096::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71642::DEBUG::2014-09-25 18:03:29,097::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71642::DEBUG::2014-09-25 18:03:29,097::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71642::DEBUG::2014-09-25 18:03:29,097::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71642::DEBUG::2014-09-25 18:03:29,098::task::993::Storage.TaskManager.Task::(_decref) Task=`04bd2f5e-5d5c-4d58-b5a5-7c080a8a8ab4`::ref 0 aborting False >Thread-71642::DEBUG::2014-09-25 18:03:29,098::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71642::DEBUG::2014-09-25 18:03:29,099::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:30,044::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:30,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0222574 s, 46.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:31,930::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:31,932::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71643::DEBUG::2014-09-25 18:03:31,935::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:32,100::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:32,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182005 s, 56.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:32,622::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:32,623::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:32,623::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:32,623::__init__::232::IOProcess::(_processLogs) (110318) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:32,629::__init__::232::IOProcess::(_processLogs) (110318) Queuing response >Thread-79::DEBUG::2014-09-25 18:03:32,630::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:03:32,642::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00136595 s, 411 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:32,643::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:32,643::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:32,643::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:32,644::__init__::232::IOProcess::(_processLogs) (110319) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:32,649::__init__::232::IOProcess::(_processLogs) (110319) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:32,651::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:32,651::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:32,651::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:32,651::__init__::232::IOProcess::(_processLogs) (110320) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:32,652::__init__::232::IOProcess::(_processLogs) (110320) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:32,653::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:32,653::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:32,653::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:32,654::__init__::232::IOProcess::(_processLogs) (110321) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:32,654::__init__::232::IOProcess::(_processLogs) (110321) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:32,655::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:32,656::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:32,656::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:32,656::__init__::232::IOProcess::(_processLogs) (110322) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:32,656::__init__::232::IOProcess::(_processLogs) (110322) Queuing response >Dummy-148::DEBUG::2014-09-25 18:03:34,148::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:34,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0376791 s, 27.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:34,947::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:34,948::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71644::DEBUG::2014-09-25 18:03:34,950::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:36,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:36,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203845 s, 50.2 MB/s\n'; <rc> = 0 >Thread-71607::WARNING::2014-09-25 18:03:36,380::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-29::DEBUG::2014-09-25 18:03:36,617::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:36,617::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:36,618::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:36,618::__init__::232::IOProcess::(_processLogs) (110323) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:36,620::__init__::232::IOProcess::(_processLogs) (110323) Queuing response >Thread-81::DEBUG::2014-09-25 18:03:36,621::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:03:36,646::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00413725 s, 89.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:36,648::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:36,648::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:36,648::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:36,648::__init__::232::IOProcess::(_processLogs) (110324) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:36,651::__init__::232::IOProcess::(_processLogs) (110324) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:37,964::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:37,966::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71645::DEBUG::2014-09-25 18:03:37,968::task::595::Storage.TaskManager.Task::(_updateState) Task=`58595681-26bd-4d83-ac7c-35f46c421809`::moving from state init -> state preparing >Thread-71645::INFO::2014-09-25 18:03:37,969::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71645::INFO::2014-09-25 18:03:37,969::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00136595', 'lastCheck': '5.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '70.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00413725', 'lastCheck': '1.3', 'valid': True}} >Thread-71645::DEBUG::2014-09-25 18:03:37,970::task::1191::Storage.TaskManager.Task::(prepare) Task=`58595681-26bd-4d83-ac7c-35f46c421809`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00136595', 'lastCheck': '5.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '70.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00413725', 'lastCheck': '1.3', 'valid': True}} >Thread-71645::DEBUG::2014-09-25 18:03:37,970::task::595::Storage.TaskManager.Task::(_updateState) Task=`58595681-26bd-4d83-ac7c-35f46c421809`::moving from state preparing -> state finished >Thread-71645::DEBUG::2014-09-25 18:03:37,970::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71645::DEBUG::2014-09-25 18:03:37,970::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71645::DEBUG::2014-09-25 18:03:37,970::task::993::Storage.TaskManager.Task::(_decref) Task=`58595681-26bd-4d83-ac7c-35f46c421809`::ref 0 aborting False >Thread-71645::DEBUG::2014-09-25 18:03:37,973::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-80::ERROR::2014-09-25 18:03:37,982::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/fileSD.py", line 587, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/outOfProcess.py", line 244, in statvfs > return self._iop.statvfs(path) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 378, in statvfs > resdict = self._sendCommand("statvfs", {"path": path}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:37,993::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:37,994::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71646::DEBUG::2014-09-25 18:03:37,996::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:38,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:38,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0267134 s, 38.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:39,126::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:39,130::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71647::DEBUG::2014-09-25 18:03:39,130::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71647::DEBUG::2014-09-25 18:03:39,136::task::595::Storage.TaskManager.Task::(_updateState) Task=`aa23fc6e-c28c-41e1-8a72-69d4b2a41963`::moving from state init -> state preparing >Thread-71647::INFO::2014-09-25 18:03:39,137::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71647::INFO::2014-09-25 18:03:39,139::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71647::DEBUG::2014-09-25 18:03:39,141::task::1191::Storage.TaskManager.Task::(prepare) Task=`aa23fc6e-c28c-41e1-8a72-69d4b2a41963`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71647::DEBUG::2014-09-25 18:03:39,142::task::595::Storage.TaskManager.Task::(_updateState) Task=`aa23fc6e-c28c-41e1-8a72-69d4b2a41963`::moving from state preparing -> state finished >Thread-71647::DEBUG::2014-09-25 18:03:39,143::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71647::DEBUG::2014-09-25 18:03:39,144::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71647::DEBUG::2014-09-25 18:03:39,145::task::993::Storage.TaskManager.Task::(_decref) Task=`aa23fc6e-c28c-41e1-8a72-69d4b2a41963`::ref 0 aborting False >Thread-71647::DEBUG::2014-09-25 18:03:39,146::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71647::DEBUG::2014-09-25 18:03:39,146::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:39,166::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71648::DEBUG::2014-09-25 18:03:39,167::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:03:39,167::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71648::DEBUG::2014-09-25 18:03:39,169::task::595::Storage.TaskManager.Task::(_updateState) Task=`395c77e9-d1c9-4f12-8ad0-5e2ac4706c30`::moving from state init -> state preparing >Thread-71648::INFO::2014-09-25 18:03:39,170::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71648::DEBUG::2014-09-25 18:03:39,170::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a6040a6c-40b3-478b-b4dc-552718928c2f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71648::DEBUG::2014-09-25 18:03:39,171::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71648::DEBUG::2014-09-25 18:03:39,171::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71648::DEBUG::2014-09-25 18:03:39,171::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`a6040a6c-40b3-478b-b4dc-552718928c2f`::Granted request >Thread-71648::DEBUG::2014-09-25 18:03:39,172::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`395c77e9-d1c9-4f12-8ad0-5e2ac4706c30`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71648::DEBUG::2014-09-25 18:03:39,172::task::993::Storage.TaskManager.Task::(_decref) Task=`395c77e9-d1c9-4f12-8ad0-5e2ac4706c30`::ref 1 aborting False >Thread-71648::INFO::2014-09-25 18:03:39,176::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71648::DEBUG::2014-09-25 18:03:39,177::task::1191::Storage.TaskManager.Task::(prepare) Task=`395c77e9-d1c9-4f12-8ad0-5e2ac4706c30`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71648::DEBUG::2014-09-25 18:03:39,177::task::595::Storage.TaskManager.Task::(_updateState) Task=`395c77e9-d1c9-4f12-8ad0-5e2ac4706c30`::moving from state preparing -> state finished >Thread-71648::DEBUG::2014-09-25 18:03:39,177::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71648::DEBUG::2014-09-25 18:03:39,177::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71648::DEBUG::2014-09-25 18:03:39,178::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71648::DEBUG::2014-09-25 18:03:39,178::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71648::DEBUG::2014-09-25 18:03:39,178::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71648::DEBUG::2014-09-25 18:03:39,179::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71648::DEBUG::2014-09-25 18:03:39,179::task::993::Storage.TaskManager.Task::(_decref) Task=`395c77e9-d1c9-4f12-8ad0-5e2ac4706c30`::ref 0 aborting False >Thread-71648::DEBUG::2014-09-25 18:03:39,179::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71648::DEBUG::2014-09-25 18:03:39,180::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:40,330::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:40,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142274 s, 72.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:41,048::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:41,049::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71649::DEBUG::2014-09-25 18:03:41,051::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:42,375::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:42,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0253487 s, 40.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:42,657::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:42,657::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:42,657::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:42,658::__init__::232::IOProcess::(_processLogs) (110325) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:42,658::__init__::232::IOProcess::(_processLogs) (110325) Queuing response >Thread-79::DEBUG::2014-09-25 18:03:42,659::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:03:42,688::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0161243 s, 34.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:42,689::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:42,689::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:42,689::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:42,689::__init__::232::IOProcess::(_processLogs) (110326) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:42,696::__init__::232::IOProcess::(_processLogs) (110326) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:42,698::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:42,698::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:42,699::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:42,699::__init__::232::IOProcess::(_processLogs) (110327) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:42,699::__init__::232::IOProcess::(_processLogs) (110327) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:42,700::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:42,701::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:42,701::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:42,701::__init__::232::IOProcess::(_processLogs) (110328) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:42,702::__init__::232::IOProcess::(_processLogs) (110328) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:42,702::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:42,703::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:42,704::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:42,704::__init__::232::IOProcess::(_processLogs) (110329) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:42,704::__init__::232::IOProcess::(_processLogs) (110329) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:44,067::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:44,068::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71650::DEBUG::2014-09-25 18:03:44,070::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:44,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:44,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136701 s, 74.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:03:46,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:46,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0319644 s, 32.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:46,654::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:46,656::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:46,657::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:46,659::__init__::232::IOProcess::(_processLogs) (110330) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:46,661::__init__::232::IOProcess::(_processLogs) (110330) Queuing response >Thread-81::DEBUG::2014-09-25 18:03:46,664::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:03:46,674::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000285288 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:46,677::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:46,679::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:46,682::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:46,684::__init__::232::IOProcess::(_processLogs) (110331) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:46,686::__init__::232::IOProcess::(_processLogs) (110331) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:47,092::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:47,093::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71651::DEBUG::2014-09-25 18:03:47,095::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:03:47,983::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:47,983::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:47,984::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:47,985::__init__::232::IOProcess::(_processLogs) (110332) Got request for method 'statvfs' >Dummy-148::DEBUG::2014-09-25 18:03:48,722::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:48,744::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133801 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:49,204::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71652::DEBUG::2014-09-25 18:03:49,205::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:03:49,205::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71652::DEBUG::2014-09-25 18:03:49,208::task::595::Storage.TaskManager.Task::(_updateState) Task=`b23bde6f-4bf1-4184-b1ff-e31253858a71`::moving from state init -> state preparing >Thread-71652::INFO::2014-09-25 18:03:49,208::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71652::INFO::2014-09-25 18:03:49,209::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71652::DEBUG::2014-09-25 18:03:49,209::task::1191::Storage.TaskManager.Task::(prepare) Task=`b23bde6f-4bf1-4184-b1ff-e31253858a71`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71652::DEBUG::2014-09-25 18:03:49,209::task::595::Storage.TaskManager.Task::(_updateState) Task=`b23bde6f-4bf1-4184-b1ff-e31253858a71`::moving from state preparing -> state finished >Thread-71652::DEBUG::2014-09-25 18:03:49,209::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71652::DEBUG::2014-09-25 18:03:49,209::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71652::DEBUG::2014-09-25 18:03:49,210::task::993::Storage.TaskManager.Task::(_decref) Task=`b23bde6f-4bf1-4184-b1ff-e31253858a71`::ref 0 aborting False >Thread-71652::DEBUG::2014-09-25 18:03:49,210::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71652::DEBUG::2014-09-25 18:03:49,210::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:49,228::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:49,229::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71653::DEBUG::2014-09-25 18:03:49,229::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71653::DEBUG::2014-09-25 18:03:49,231::task::595::Storage.TaskManager.Task::(_updateState) Task=`c519a569-a51b-430f-a281-1724de83476b`::moving from state init -> state preparing >Thread-71653::INFO::2014-09-25 18:03:49,231::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71653::DEBUG::2014-09-25 18:03:49,232::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cdf9aebe-0465-4f35-a184-eb8dc4ecb0ad`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71653::DEBUG::2014-09-25 18:03:49,232::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71653::DEBUG::2014-09-25 18:03:49,232::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71653::DEBUG::2014-09-25 18:03:49,233::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`cdf9aebe-0465-4f35-a184-eb8dc4ecb0ad`::Granted request >Thread-71653::DEBUG::2014-09-25 18:03:49,233::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`c519a569-a51b-430f-a281-1724de83476b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71653::DEBUG::2014-09-25 18:03:49,233::task::993::Storage.TaskManager.Task::(_decref) Task=`c519a569-a51b-430f-a281-1724de83476b`::ref 1 aborting False >Thread-71653::INFO::2014-09-25 18:03:49,238::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71653::DEBUG::2014-09-25 18:03:49,238::task::1191::Storage.TaskManager.Task::(prepare) Task=`c519a569-a51b-430f-a281-1724de83476b`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71653::DEBUG::2014-09-25 18:03:49,238::task::595::Storage.TaskManager.Task::(_updateState) Task=`c519a569-a51b-430f-a281-1724de83476b`::moving from state preparing -> state finished >Thread-71653::DEBUG::2014-09-25 18:03:49,239::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71653::DEBUG::2014-09-25 18:03:49,239::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71653::DEBUG::2014-09-25 18:03:49,239::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71653::DEBUG::2014-09-25 18:03:49,240::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71653::DEBUG::2014-09-25 18:03:49,240::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71653::DEBUG::2014-09-25 18:03:49,240::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71653::DEBUG::2014-09-25 18:03:49,240::task::993::Storage.TaskManager.Task::(_decref) Task=`c519a569-a51b-430f-a281-1724de83476b`::ref 0 aborting False >Thread-71653::DEBUG::2014-09-25 18:03:49,241::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71653::DEBUG::2014-09-25 18:03:49,241::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:50,121::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:50,122::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71654::DEBUG::2014-09-25 18:03:50,124::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:50,765::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:51,040::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.264208 s, 3.9 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 18:03:52,704::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 18:03:52,708::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,709::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,709::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,709::__init__::232::IOProcess::(_processLogs) (110333) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:03:52,710::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,710::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,710::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,711::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,711::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,711::__init__::232::IOProcess::(_processLogs) (110334) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:03:52,711::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,712::__init__::232::IOProcess::(_processLogs) (110335) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:03:52,716::__init__::232::IOProcess::(_processLogs) (110333) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:52,723::__init__::232::IOProcess::(_processLogs) (110335) Queuing response >Thread-79::DEBUG::2014-09-25 18:03:52,727::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 18:03:52,728::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,728::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,728::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,729::__init__::232::IOProcess::(_processLogs) (110336) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 18:03:52,770::__init__::232::IOProcess::(_processLogs) (110336) Queuing response >Thread-79::DEBUG::2014-09-25 18:03:52,771::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:03:52,772::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,773::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,773::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,773::__init__::232::IOProcess::(_processLogs) (110337) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 18:03:52,783::__init__::232::IOProcess::(_processLogs) (110337) Queuing response >Thread-79::DEBUG::2014-09-25 18:03:52,783::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 18:03:52,784::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,785::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,785::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,786::__init__::232::IOProcess::(_processLogs) (110338) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:52,786::__init__::232::IOProcess::(_processLogs) (110338) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:52,787::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,787::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,787::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,787::__init__::232::IOProcess::(_processLogs) (110339) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:03:52,799::__init__::232::IOProcess::(_processLogs) (110339) Queuing response >Thread-79::DEBUG::2014-09-25 18:03:52,799::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 18:03:52,800::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 18:03:52,800::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 18:03:52,801::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,801::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,802::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,802::__init__::232::IOProcess::(_processLogs) (110340) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:52,809::__init__::232::IOProcess::(_processLogs) (110340) Queuing response >Thread-79::DEBUG::2014-09-25 18:03:52,810::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:03:52,827::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00831461 s, 67.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:52,828::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,828::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,828::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,829::__init__::232::IOProcess::(_processLogs) (110341) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:52,842::__init__::232::IOProcess::(_processLogs) (110341) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:52,843::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,843::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,843::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,844::__init__::232::IOProcess::(_processLogs) (110342) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:52,849::__init__::232::IOProcess::(_processLogs) (110342) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:52,850::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,850::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,850::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,850::__init__::232::IOProcess::(_processLogs) (110343) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:52,856::__init__::232::IOProcess::(_processLogs) (110343) Queuing response >Thread-29::DEBUG::2014-09-25 18:03:52,857::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:52,857::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:52,858::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:52,858::__init__::232::IOProcess::(_processLogs) (110344) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:03:52,859::__init__::232::IOProcess::(_processLogs) (110344) Queuing response >Dummy-148::DEBUG::2014-09-25 18:03:53,059::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:53,138::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:53,140::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71658::DEBUG::2014-09-25 18:03:53,143::task::595::Storage.TaskManager.Task::(_updateState) Task=`d865513f-a599-4153-96a8-926e7a911f4c`::moving from state init -> state preparing >Thread-71658::INFO::2014-09-25 18:03:53,143::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71658::INFO::2014-09-25 18:03:53,143::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00831461', 'lastCheck': '0.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '15.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000285288', 'lastCheck': '6.5', 'valid': True}} >Thread-71658::DEBUG::2014-09-25 18:03:53,144::task::1191::Storage.TaskManager.Task::(prepare) Task=`d865513f-a599-4153-96a8-926e7a911f4c`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00831461', 'lastCheck': '0.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '15.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000285288', 'lastCheck': '6.5', 'valid': True}} >Thread-71658::DEBUG::2014-09-25 18:03:53,144::task::595::Storage.TaskManager.Task::(_updateState) Task=`d865513f-a599-4153-96a8-926e7a911f4c`::moving from state preparing -> state finished >Thread-71658::DEBUG::2014-09-25 18:03:53,144::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71658::DEBUG::2014-09-25 18:03:53,144::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71658::DEBUG::2014-09-25 18:03:53,145::task::993::Storage.TaskManager.Task::(_decref) Task=`d865513f-a599-4153-96a8-926e7a911f4c`::ref 0 aborting False >Thread-71658::DEBUG::2014-09-25 18:03:53,147::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:53,180::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:53,180::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71659::DEBUG::2014-09-25 18:03:53,182::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:53,285::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.213305 s, 4.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:03:55,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:55,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0188429 s, 54.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:56,245::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:56,246::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71660::DEBUG::2014-09-25 18:03:56,248::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:03:56,685::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:56,685::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:56,685::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:56,686::__init__::232::IOProcess::(_processLogs) (110345) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:56,686::__init__::232::IOProcess::(_processLogs) (110345) Queuing response >Thread-81::DEBUG::2014-09-25 18:03:56,687::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:03:56,697::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000389732 s, 952 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:03:56,698::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:03:56,698::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:03:56,699::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:03:56,700::__init__::232::IOProcess::(_processLogs) (110346) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:03:56,700::__init__::232::IOProcess::(_processLogs) (110346) Queuing response >Dummy-148::DEBUG::2014-09-25 18:03:57,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:57,385::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211249 s, 48.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:59,264::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:59,265::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71661::DEBUG::2014-09-25 18:03:59,267::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:59,268::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:59,269::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71662::DEBUG::2014-09-25 18:03:59,269::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71662::DEBUG::2014-09-25 18:03:59,271::task::595::Storage.TaskManager.Task::(_updateState) Task=`3163ef76-2fba-4108-9a48-279cc4daba2e`::moving from state init -> state preparing >Thread-71662::INFO::2014-09-25 18:03:59,272::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71662::INFO::2014-09-25 18:03:59,272::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71662::DEBUG::2014-09-25 18:03:59,272::task::1191::Storage.TaskManager.Task::(prepare) Task=`3163ef76-2fba-4108-9a48-279cc4daba2e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71662::DEBUG::2014-09-25 18:03:59,273::task::595::Storage.TaskManager.Task::(_updateState) Task=`3163ef76-2fba-4108-9a48-279cc4daba2e`::moving from state preparing -> state finished >Thread-71662::DEBUG::2014-09-25 18:03:59,273::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71662::DEBUG::2014-09-25 18:03:59,273::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71662::DEBUG::2014-09-25 18:03:59,273::task::993::Storage.TaskManager.Task::(_decref) Task=`3163ef76-2fba-4108-9a48-279cc4daba2e`::ref 0 aborting False >Thread-71662::DEBUG::2014-09-25 18:03:59,274::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71662::DEBUG::2014-09-25 18:03:59,274::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:03:59,289::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:03:59,290::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71663::DEBUG::2014-09-25 18:03:59,290::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71663::DEBUG::2014-09-25 18:03:59,292::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f9077bd-6c01-45ef-915b-1cce8abd2b02`::moving from state init -> state preparing >Thread-71663::INFO::2014-09-25 18:03:59,292::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71663::DEBUG::2014-09-25 18:03:59,293::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`786ab630-fca8-4a88-a140-38a9552bca9f`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71663::DEBUG::2014-09-25 18:03:59,293::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71663::DEBUG::2014-09-25 18:03:59,294::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71663::DEBUG::2014-09-25 18:03:59,294::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`786ab630-fca8-4a88-a140-38a9552bca9f`::Granted request >Thread-71663::DEBUG::2014-09-25 18:03:59,295::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`9f9077bd-6c01-45ef-915b-1cce8abd2b02`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71663::DEBUG::2014-09-25 18:03:59,295::task::993::Storage.TaskManager.Task::(_decref) Task=`9f9077bd-6c01-45ef-915b-1cce8abd2b02`::ref 1 aborting False >Thread-71663::INFO::2014-09-25 18:03:59,299::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71663::DEBUG::2014-09-25 18:03:59,299::task::1191::Storage.TaskManager.Task::(prepare) Task=`9f9077bd-6c01-45ef-915b-1cce8abd2b02`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71663::DEBUG::2014-09-25 18:03:59,300::task::595::Storage.TaskManager.Task::(_updateState) Task=`9f9077bd-6c01-45ef-915b-1cce8abd2b02`::moving from state preparing -> state finished >Thread-71663::DEBUG::2014-09-25 18:03:59,300::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71663::DEBUG::2014-09-25 18:03:59,300::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71663::DEBUG::2014-09-25 18:03:59,300::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71663::DEBUG::2014-09-25 18:03:59,301::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71663::DEBUG::2014-09-25 18:03:59,301::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71663::DEBUG::2014-09-25 18:03:59,301::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71663::DEBUG::2014-09-25 18:03:59,302::task::993::Storage.TaskManager.Task::(_decref) Task=`9f9077bd-6c01-45ef-915b-1cce8abd2b02`::ref 0 aborting False >Thread-71663::DEBUG::2014-09-25 18:03:59,302::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71663::DEBUG::2014-09-25 18:03:59,303::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:03:59,404::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:03:59,429::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138421 s, 74.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:01,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:01,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0196572 s, 52.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:04:02,286::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:04:02,287::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71664::DEBUG::2014-09-25 18:04:02,289::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:04:02,861::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:02,861::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:02,862::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:02,862::__init__::232::IOProcess::(_processLogs) (110347) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:02,864::__init__::232::IOProcess::(_processLogs) (110347) Queuing response >Thread-79::DEBUG::2014-09-25 18:04:02,865::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:04:02,897::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0187728 s, 29.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:02,899::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:02,899::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:02,899::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:02,899::__init__::232::IOProcess::(_processLogs) (110348) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:02,902::__init__::232::IOProcess::(_processLogs) (110348) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:02,903::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:02,904::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:02,904::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:02,905::__init__::232::IOProcess::(_processLogs) (110349) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:02,905::__init__::232::IOProcess::(_processLogs) (110349) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:02,906::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:02,906::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:02,907::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:02,907::__init__::232::IOProcess::(_processLogs) (110350) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:02,907::__init__::232::IOProcess::(_processLogs) (110350) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:02,908::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:02,908::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:02,909::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:02,909::__init__::232::IOProcess::(_processLogs) (110351) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:02,910::__init__::232::IOProcess::(_processLogs) (110351) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:03,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:03,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133586 s, 76.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:04:05,304::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:04:05,306::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71665::DEBUG::2014-09-25 18:04:05,308::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:04:05,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:05,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184535 s, 55.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:06,701::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:06,701::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:06,701::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:06,702::__init__::232::IOProcess::(_processLogs) (110352) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:06,711::__init__::232::IOProcess::(_processLogs) (110352) Queuing response >Thread-81::DEBUG::2014-09-25 18:04:06,712::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:04:06,738::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00435714 s, 85.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:06,739::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:06,740::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:06,740::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:06,740::__init__::232::IOProcess::(_processLogs) (110353) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:06,745::__init__::232::IOProcess::(_processLogs) (110353) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:07,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:07,621::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199713 s, 51.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:04:08,320::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:04:08,321::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71666::DEBUG::2014-09-25 18:04:08,324::task::595::Storage.TaskManager.Task::(_updateState) Task=`86d7d7bf-967f-4244-8015-fe5d7644aa86`::moving from state init -> state preparing >Thread-71666::INFO::2014-09-25 18:04:08,324::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71666::INFO::2014-09-25 18:04:08,325::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0187728', 'lastCheck': '5.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '30.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00435714', 'lastCheck': '1.6', 'valid': True}} >Thread-71666::DEBUG::2014-09-25 18:04:08,325::task::1191::Storage.TaskManager.Task::(prepare) Task=`86d7d7bf-967f-4244-8015-fe5d7644aa86`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0187728', 'lastCheck': '5.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '30.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00435714', 'lastCheck': '1.6', 'valid': True}} >Thread-71666::DEBUG::2014-09-25 18:04:08,325::task::595::Storage.TaskManager.Task::(_updateState) Task=`86d7d7bf-967f-4244-8015-fe5d7644aa86`::moving from state preparing -> state finished >Thread-71666::DEBUG::2014-09-25 18:04:08,325::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71666::DEBUG::2014-09-25 18:04:08,326::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71666::DEBUG::2014-09-25 18:04:08,326::task::993::Storage.TaskManager.Task::(_decref) Task=`86d7d7bf-967f-4244-8015-fe5d7644aa86`::ref 0 aborting False >Thread-71666::DEBUG::2014-09-25 18:04:08,328::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:04:08,351::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:04:08,354::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71667::DEBUG::2014-09-25 18:04:08,357::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:04:09,327::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71668::DEBUG::2014-09-25 18:04:09,328::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:04:09,328::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71668::DEBUG::2014-09-25 18:04:09,330::task::595::Storage.TaskManager.Task::(_updateState) Task=`28dcf68b-35e1-47f2-9f94-ea47c9fae2f9`::moving from state init -> state preparing >Thread-71668::INFO::2014-09-25 18:04:09,331::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71668::INFO::2014-09-25 18:04:09,331::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71668::DEBUG::2014-09-25 18:04:09,332::task::1191::Storage.TaskManager.Task::(prepare) Task=`28dcf68b-35e1-47f2-9f94-ea47c9fae2f9`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71668::DEBUG::2014-09-25 18:04:09,332::task::595::Storage.TaskManager.Task::(_updateState) Task=`28dcf68b-35e1-47f2-9f94-ea47c9fae2f9`::moving from state preparing -> state finished >Thread-71668::DEBUG::2014-09-25 18:04:09,332::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71668::DEBUG::2014-09-25 18:04:09,332::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71668::DEBUG::2014-09-25 18:04:09,332::task::993::Storage.TaskManager.Task::(_decref) Task=`28dcf68b-35e1-47f2-9f94-ea47c9fae2f9`::ref 0 aborting False >Thread-71668::DEBUG::2014-09-25 18:04:09,333::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71668::DEBUG::2014-09-25 18:04:09,333::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:04:09,343::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:04:09,345::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71669::DEBUG::2014-09-25 18:04:09,345::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71669::DEBUG::2014-09-25 18:04:09,346::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ead3138-5d38-44ad-801a-d80e435b8cde`::moving from state init -> state preparing >Thread-71669::INFO::2014-09-25 18:04:09,346::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71669::DEBUG::2014-09-25 18:04:09,347::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`708ede1d-cc16-40ed-8c20-c19e884056da`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71669::DEBUG::2014-09-25 18:04:09,347::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71669::DEBUG::2014-09-25 18:04:09,348::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71669::DEBUG::2014-09-25 18:04:09,348::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`708ede1d-cc16-40ed-8c20-c19e884056da`::Granted request >Thread-71669::DEBUG::2014-09-25 18:04:09,349::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`3ead3138-5d38-44ad-801a-d80e435b8cde`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71669::DEBUG::2014-09-25 18:04:09,349::task::993::Storage.TaskManager.Task::(_decref) Task=`3ead3138-5d38-44ad-801a-d80e435b8cde`::ref 1 aborting False >Thread-71669::INFO::2014-09-25 18:04:09,353::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71669::DEBUG::2014-09-25 18:04:09,353::task::1191::Storage.TaskManager.Task::(prepare) Task=`3ead3138-5d38-44ad-801a-d80e435b8cde`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71669::DEBUG::2014-09-25 18:04:09,354::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ead3138-5d38-44ad-801a-d80e435b8cde`::moving from state preparing -> state finished >Thread-71669::DEBUG::2014-09-25 18:04:09,354::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71669::DEBUG::2014-09-25 18:04:09,354::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71669::DEBUG::2014-09-25 18:04:09,354::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71669::DEBUG::2014-09-25 18:04:09,355::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71669::DEBUG::2014-09-25 18:04:09,355::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71669::DEBUG::2014-09-25 18:04:09,355::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71669::DEBUG::2014-09-25 18:04:09,356::task::993::Storage.TaskManager.Task::(_decref) Task=`3ead3138-5d38-44ad-801a-d80e435b8cde`::ref 0 aborting False >Thread-71669::DEBUG::2014-09-25 18:04:09,356::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71669::DEBUG::2014-09-25 18:04:09,356::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:04:09,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:09,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175586 s, 58.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:04:11,425::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:04:11,427::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71670::DEBUG::2014-09-25 18:04:11,428::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:04:11,689::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:11,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201442 s, 50.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:12,910::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:12,910::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:12,910::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:12,911::__init__::232::IOProcess::(_processLogs) (110354) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:12,918::__init__::232::IOProcess::(_processLogs) (110354) Queuing response >Thread-79::DEBUG::2014-09-25 18:04:12,919::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:04:12,946::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00857811 s, 65.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:12,947::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:12,948::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:12,948::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:12,948::__init__::232::IOProcess::(_processLogs) (110355) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:12,950::__init__::232::IOProcess::(_processLogs) (110355) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:12,952::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:12,952::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:12,953::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:12,953::__init__::232::IOProcess::(_processLogs) (110356) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:12,953::__init__::232::IOProcess::(_processLogs) (110356) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:12,954::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:12,954::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:12,955::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:12,955::__init__::232::IOProcess::(_processLogs) (110357) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:12,955::__init__::232::IOProcess::(_processLogs) (110357) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:12,956::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:12,956::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:12,957::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:12,957::__init__::232::IOProcess::(_processLogs) (110358) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:12,958::__init__::232::IOProcess::(_processLogs) (110358) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:13,738::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:13,761::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135802 s, 75.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:04:14,444::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:04:14,445::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71671::DEBUG::2014-09-25 18:04:14,447::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:04:15,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:15,812::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199575 s, 51.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:16,748::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:16,749::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:16,749::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:16,750::__init__::232::IOProcess::(_processLogs) (110359) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:16,753::__init__::232::IOProcess::(_processLogs) (110359) Queuing response >Thread-81::DEBUG::2014-09-25 18:04:16,755::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:04:16,774::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00879347 s, 42.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:16,776::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:16,777::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:16,778::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:16,778::__init__::232::IOProcess::(_processLogs) (110360) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:16,781::__init__::232::IOProcess::(_processLogs) (110360) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:17,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:17,866::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205635 s, 49.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:19,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:19,911::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160709 s, 63.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:21,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:21,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013346 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:22,958::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:22,958::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:22,959::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:22,959::__init__::232::IOProcess::(_processLogs) (110361) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:22,966::__init__::232::IOProcess::(_processLogs) (110361) Queuing response >Thread-79::DEBUG::2014-09-25 18:04:22,967::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:04:22,982::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00205837 s, 273 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:22,984::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:22,984::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:22,984::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:22,984::__init__::232::IOProcess::(_processLogs) (110362) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:22,991::__init__::232::IOProcess::(_processLogs) (110362) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:22,993::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:22,994::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:22,994::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:22,994::__init__::232::IOProcess::(_processLogs) (110363) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:22,995::__init__::232::IOProcess::(_processLogs) (110363) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:22,995::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:22,996::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:22,996::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:22,997::__init__::232::IOProcess::(_processLogs) (110364) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:22,997::__init__::232::IOProcess::(_processLogs) (110364) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:22,997::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:22,998::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:22,999::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:22,999::__init__::232::IOProcess::(_processLogs) (110365) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:22,999::__init__::232::IOProcess::(_processLogs) (110365) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:23,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:24,014::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0225466 s, 45.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:26,033::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:26,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166377 s, 61.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:26,784::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:26,784::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:26,784::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:26,785::__init__::232::IOProcess::(_processLogs) (110366) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:26,788::__init__::232::IOProcess::(_processLogs) (110366) Queuing response >Thread-81::DEBUG::2014-09-25 18:04:26,789::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:04:26,808::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00440146 s, 84.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:26,809::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:26,810::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:26,810::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:26,810::__init__::232::IOProcess::(_processLogs) (110367) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:26,811::__init__::232::IOProcess::(_processLogs) (110367) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:28,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:28,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159709 s, 64.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:30,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:30,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0267291 s, 38.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:32,190::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:32,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0397887 s, 25.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:33,000::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:33,000::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:33,001::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:33,001::__init__::232::IOProcess::(_processLogs) (110368) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:33,005::__init__::232::IOProcess::(_processLogs) (110368) Queuing response >Thread-79::DEBUG::2014-09-25 18:04:33,006::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:04:33,026::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00882916 s, 63.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:33,028::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:33,028::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:33,028::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:33,028::__init__::232::IOProcess::(_processLogs) (110369) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:33,032::__init__::232::IOProcess::(_processLogs) (110369) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:33,034::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:33,034::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:33,034::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:33,035::__init__::232::IOProcess::(_processLogs) (110370) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:33,035::__init__::232::IOProcess::(_processLogs) (110370) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:33,036::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:33,036::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:33,037::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:33,037::__init__::232::IOProcess::(_processLogs) (110371) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:33,037::__init__::232::IOProcess::(_processLogs) (110371) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:33,038::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:33,038::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:33,039::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:33,040::__init__::232::IOProcess::(_processLogs) (110372) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:33,040::__init__::232::IOProcess::(_processLogs) (110372) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:34,269::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:34,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148432 s, 69.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:36,313::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:36,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133291 s, 76.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:36,813::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:36,814::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:36,814::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:36,814::__init__::232::IOProcess::(_processLogs) (110373) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:36,815::__init__::232::IOProcess::(_processLogs) (110373) Queuing response >Thread-81::DEBUG::2014-09-25 18:04:36,816::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:04:36,833::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00368096 s, 101 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:36,834::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:36,835::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:36,835::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:36,835::__init__::232::IOProcess::(_processLogs) (110374) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:36,838::__init__::232::IOProcess::(_processLogs) (110374) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:38,355::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:38,388::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0216694 s, 47.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:40,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:40,438::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204899 s, 50.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:42,458::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:42,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0337019 s, 30.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:43,041::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:43,041::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:43,041::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:43,041::__init__::232::IOProcess::(_processLogs) (110375) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:43,047::__init__::232::IOProcess::(_processLogs) (110375) Queuing response >Thread-79::DEBUG::2014-09-25 18:04:43,048::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:04:43,065::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00624572 s, 90.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:43,067::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:43,067::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:43,067::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:43,067::__init__::232::IOProcess::(_processLogs) (110376) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:43,068::__init__::232::IOProcess::(_processLogs) (110376) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:43,069::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:43,069::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:43,070::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:43,070::__init__::232::IOProcess::(_processLogs) (110377) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:43,071::__init__::232::IOProcess::(_processLogs) (110377) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:43,071::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:43,071::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:43,072::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:43,073::__init__::232::IOProcess::(_processLogs) (110378) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:43,073::__init__::232::IOProcess::(_processLogs) (110378) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:43,073::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:43,073::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:43,074::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:43,075::__init__::232::IOProcess::(_processLogs) (110379) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:43,075::__init__::232::IOProcess::(_processLogs) (110379) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:44,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:44,573::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192525 s, 53.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:46,594::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:46,622::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207324 s, 49.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:46,841::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:46,842::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:46,843::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:46,844::__init__::232::IOProcess::(_processLogs) (110380) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:46,845::__init__::232::IOProcess::(_processLogs) (110380) Queuing response >Thread-81::DEBUG::2014-09-25 18:04:46,847::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:04:46,863::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00339402 s, 109 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:46,865::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:46,866::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:46,867::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:46,868::__init__::232::IOProcess::(_processLogs) (110381) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:46,870::__init__::232::IOProcess::(_processLogs) (110381) Queuing response >Thread-80::ERROR::2014-09-25 18:04:47,983::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/fileSD.py", line 587, in selftest > self.oop.os.statvfs(self.domaindir) > File "/usr/share/vdsm/storage/outOfProcess.py", line 244, in statvfs > return self._iop.statvfs(path) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 378, in statvfs > resdict = self._sendCommand("statvfs", {"path": path}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Dummy-148::DEBUG::2014-09-25 18:04:48,728::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:48,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218726 s, 46.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:04:50,781::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:50,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0566784 s, 18.1 MB/s\n'; <rc> = 0 >Thread-71656::WARNING::2014-09-25 18:04:52,708::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Dummy-148::DEBUG::2014-09-25 18:04:52,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:52,899::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137851 s, 74.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:53,076::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:53,076::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:53,077::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:53,077::__init__::232::IOProcess::(_processLogs) (110382) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:53,081::__init__::232::IOProcess::(_processLogs) (110382) Queuing response >Thread-79::DEBUG::2014-09-25 18:04:53,083::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:04:53,116::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00857733 s, 65.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:53,117::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:53,118::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:53,118::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:53,118::__init__::232::IOProcess::(_processLogs) (110383) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:53,142::__init__::232::IOProcess::(_processLogs) (110383) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:53,143::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:53,143::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:53,144::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:53,144::__init__::232::IOProcess::(_processLogs) (110384) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:53,151::__init__::232::IOProcess::(_processLogs) (110384) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:53,152::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:53,153::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:53,153::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:53,153::__init__::232::IOProcess::(_processLogs) (110385) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:53,169::__init__::232::IOProcess::(_processLogs) (110385) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:53,170::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:53,170::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:53,170::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:53,171::__init__::232::IOProcess::(_processLogs) (110386) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:04:53,183::__init__::232::IOProcess::(_processLogs) (110386) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:54,919::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:54,953::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020896 s, 49.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:56,873::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:56,873::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:56,873::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:56,874::__init__::232::IOProcess::(_processLogs) (110387) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:56,876::__init__::232::IOProcess::(_processLogs) (110387) Queuing response >Thread-81::DEBUG::2014-09-25 18:04:56,877::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:04:56,891::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00132807 s, 279 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:04:56,893::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:56,893::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:56,893::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:56,893::__init__::232::IOProcess::(_processLogs) (110388) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:04:56,895::__init__::232::IOProcess::(_processLogs) (110388) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:56,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:57,005::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0206908 s, 49.5 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 18:04:57,984::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 18:04:57,988::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:57,989::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:57,989::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:57,989::__init__::232::IOProcess::(_processLogs) (110389) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:04:57,989::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:57,990::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:57,990::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:04:57,990::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:57,990::__init__::232::IOProcess::(_processLogs) (110390) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:04:57,991::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:04:57,991::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:04:57,991::__init__::232::IOProcess::(_processLogs) (110391) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:04:57,993::__init__::232::IOProcess::(_processLogs) (110389) Queuing response >Thread-29::DEBUG::2014-09-25 18:04:57,995::__init__::232::IOProcess::(_processLogs) (110391) Queuing response >Dummy-148::DEBUG::2014-09-25 18:04:59,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:04:59,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132613 s, 77.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:01,066::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:01,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163049 s, 62.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:03,112::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:03,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.020227 s, 50.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:03,184::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:03,185::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:03,185::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:03,185::__init__::232::IOProcess::(_processLogs) (110392) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:03,187::__init__::232::IOProcess::(_processLogs) (110392) Queuing response >Thread-79::DEBUG::2014-09-25 18:05:03,188::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:05:03,204::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00674387 s, 83.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:03,205::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:03,205::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:03,206::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:03,206::__init__::232::IOProcess::(_processLogs) (110393) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:03,206::__init__::232::IOProcess::(_processLogs) (110393) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:03,207::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:03,208::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:03,209::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:03,209::__init__::232::IOProcess::(_processLogs) (110394) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:03,209::__init__::232::IOProcess::(_processLogs) (110394) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:03,210::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:03,210::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:03,211::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:03,211::__init__::232::IOProcess::(_processLogs) (110395) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:03,211::__init__::232::IOProcess::(_processLogs) (110395) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:03,212::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:03,212::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:03,213::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:03,213::__init__::232::IOProcess::(_processLogs) (110396) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:03,214::__init__::232::IOProcess::(_processLogs) (110396) Queuing response >Dummy-148::DEBUG::2014-09-25 18:05:05,167::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:05,198::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189334 s, 54.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:06,897::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:06,898::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:06,898::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:06,898::__init__::232::IOProcess::(_processLogs) (110397) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:06,899::__init__::232::IOProcess::(_processLogs) (110397) Queuing response >Thread-81::DEBUG::2014-09-25 18:05:06,900::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:05:06,909::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000351742 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:06,910::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:06,910::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:06,911::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:06,911::__init__::232::IOProcess::(_processLogs) (110398) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:06,911::__init__::232::IOProcess::(_processLogs) (110398) Queuing response >Dummy-148::DEBUG::2014-09-25 18:05:07,217::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:07,247::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165732 s, 61.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:09,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:09,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164405 s, 62.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:11,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:11,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0216992 s, 47.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:13,214::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:13,214::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:13,215::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:13,215::__init__::232::IOProcess::(_processLogs) (110399) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:13,216::__init__::232::IOProcess::(_processLogs) (110399) Queuing response >Thread-79::DEBUG::2014-09-25 18:05:13,216::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:05:13,226::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00049853 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:13,227::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:13,227::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:13,227::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:13,228::__init__::232::IOProcess::(_processLogs) (110400) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:13,228::__init__::232::IOProcess::(_processLogs) (110400) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:13,230::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:13,230::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:13,230::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:13,230::__init__::232::IOProcess::(_processLogs) (110401) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:13,231::__init__::232::IOProcess::(_processLogs) (110401) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:13,232::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:13,232::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:13,233::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:13,233::__init__::232::IOProcess::(_processLogs) (110402) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:13,233::__init__::232::IOProcess::(_processLogs) (110402) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:13,234::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:13,234::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:13,234::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:13,234::__init__::232::IOProcess::(_processLogs) (110403) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:13,235::__init__::232::IOProcess::(_processLogs) (110403) Queuing response >Dummy-148::DEBUG::2014-09-25 18:05:13,368::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:13,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130322 s, 78.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:15,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:15,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199458 s, 51.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:16,914::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:16,915::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:16,916::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:16,917::__init__::232::IOProcess::(_processLogs) (110404) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:16,918::__init__::232::IOProcess::(_processLogs) (110404) Queuing response >Thread-81::DEBUG::2014-09-25 18:05:16,919::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:05:16,934::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00597522 s, 62.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:16,936::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:16,936::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:16,937::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:16,938::__init__::232::IOProcess::(_processLogs) (110405) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:16,939::__init__::232::IOProcess::(_processLogs) (110405) Queuing response >Dummy-148::DEBUG::2014-09-25 18:05:17,468::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:17,493::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144495 s, 70.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:19,512::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:19,534::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131168 s, 78.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:21,554::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:21,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0268369 s, 38.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:23,236::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:23,237::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:23,237::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:23,237::__init__::232::IOProcess::(_processLogs) (110406) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:23,247::__init__::232::IOProcess::(_processLogs) (110406) Queuing response >Thread-79::DEBUG::2014-09-25 18:05:23,248::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:05:23,269::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00306675 s, 183 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:23,270::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:23,270::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:23,270::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:23,270::__init__::232::IOProcess::(_processLogs) (110407) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:23,272::__init__::232::IOProcess::(_processLogs) (110407) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:23,274::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:23,274::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:23,275::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:23,275::__init__::232::IOProcess::(_processLogs) (110408) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:23,275::__init__::232::IOProcess::(_processLogs) (110408) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:23,276::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:23,276::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:23,277::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:23,277::__init__::232::IOProcess::(_processLogs) (110409) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:23,277::__init__::232::IOProcess::(_processLogs) (110409) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:23,278::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:23,279::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:23,279::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:23,279::__init__::232::IOProcess::(_processLogs) (110410) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:23,280::__init__::232::IOProcess::(_processLogs) (110410) Queuing response >Dummy-148::DEBUG::2014-09-25 18:05:23,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:23,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150669 s, 68.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:25,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:25,695::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016294 s, 62.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:26,941::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:26,942::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:26,942::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:26,942::__init__::232::IOProcess::(_processLogs) (110411) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:26,946::__init__::232::IOProcess::(_processLogs) (110411) Queuing response >Thread-81::DEBUG::2014-09-25 18:05:26,946::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:05:26,961::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000445615 s, 833 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:26,962::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:26,963::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:26,964::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:26,964::__init__::232::IOProcess::(_processLogs) (110412) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:26,965::__init__::232::IOProcess::(_processLogs) (110412) Queuing response >Dummy-148::DEBUG::2014-09-25 18:05:27,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:27,746::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0185836 s, 55.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:29,766::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:29,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200627 s, 51.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:31,822::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:31,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148336 s, 69.0 MB/s\n'; <rc> = 0 >Detector thread::DEBUG::2014-09-25 18:05:32,019::protocoldetector::160::vds.MultiProtocolAcceptor::(_add_connection) Adding connection from 10.34.63.74:54607 >Detector thread::DEBUG::2014-09-25 18:05:32,024::protocoldetector::171::vds.MultiProtocolAcceptor::(_remove_connection) Connection removed from 10.34.63.74:54607 >Detector thread::DEBUG::2014-09-25 18:05:32,024::protocoldetector::197::vds.MultiProtocolAcceptor::(_handle_connection_read) Detected protocol stomp from 10.34.63.74:54607 >Detector thread::DEBUG::2014-09-25 18:05:32,025::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='CONNECT'> >Detector thread::INFO::2014-09-25 18:05:32,025::stompReactor::66::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request >Detector thread::DEBUG::2014-09-25 18:05:32,026::stompReactor::338::protocoldetector.StompDetector::(handleSocket) Stomp detected from ('10.34.63.74', 54607) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:32,027::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SUBSCRIBE'> >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:32,028::stompReactor::89::Broker.StompAdapter::(_cmd_subscribe) Subscribe command ignored >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:32,068::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:32,070::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71675::DEBUG::2014-09-25 18:05:32,073::task::595::Storage.TaskManager.Task::(_updateState) Task=`bebd1fe3-6202-4a2f-b6a4-f0f24f685d70`::moving from state init -> state preparing >Thread-71675::INFO::2014-09-25 18:05:32,073::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71675::INFO::2014-09-25 18:05:32,074::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00306675', 'lastCheck': '8.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.1', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000445615', 'lastCheck': '5.1', 'valid': True}} >Thread-71675::DEBUG::2014-09-25 18:05:32,074::task::1191::Storage.TaskManager.Task::(prepare) Task=`bebd1fe3-6202-4a2f-b6a4-f0f24f685d70`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00306675', 'lastCheck': '8.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.1', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000445615', 'lastCheck': '5.1', 'valid': True}} >Thread-71675::DEBUG::2014-09-25 18:05:32,074::task::595::Storage.TaskManager.Task::(_updateState) Task=`bebd1fe3-6202-4a2f-b6a4-f0f24f685d70`::moving from state preparing -> state finished >Thread-71675::DEBUG::2014-09-25 18:05:32,074::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71675::DEBUG::2014-09-25 18:05:32,074::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71675::DEBUG::2014-09-25 18:05:32,075::task::993::Storage.TaskManager.Task::(_decref) Task=`bebd1fe3-6202-4a2f-b6a4-f0f24f685d70`::ref 0 aborting False >Thread-71675::DEBUG::2014-09-25 18:05:32,077::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:32,308::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71676::DEBUG::2014-09-25 18:05:32,309::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getCapabilities' in bridge with {} >JsonRpcServer::DEBUG::2014-09-25 18:05:32,309::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71676::DEBUG::2014-09-25 18:05:32,313::utils::738::root::(execCmd) /sbin/ip route show to 0.0.0.0/0 table all (cwd None) >Thread-71676::DEBUG::2014-09-25 18:05:32,321::utils::758::root::(execCmd) SUCCESS: <err> = ''; <rc> = 0 >Thread-71676::DEBUG::2014-09-25 18:05:32,354::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getCapabilities' in bridge with {'HBAInventory': {'iSCSI': [{'InitiatorName': 'iqn.1994-05.com.redhat:3bd6acb1d4f2'}], 'FC': []}, 'packages2': {'kernel': {'release': '123.el7.x86_64', 'buildtime': 1399281417.0, 'version': '3.10.0'}, 'spice-server': {'release': '5.el7', 'buildtime': 1391017388L, 'version': '0.12.4'}, 'vdsm': {'release': '3.el7.beta', 'buildtime': 1410353965L, 'version': '4.16.3'}, 'qemu-kvm': {'release': '60.el7_0.7', 'buildtime': 1409068381L, 'version': '1.5.3'}, 'libvirt': {'release': '29.el7_0.1', 'buildtime': 1404387754L, 'version': '1.1.1'}, 'qemu-img': {'release': '60.el7_0.7', 'buildtime': 1409068381L, 'version': '1.5.3'}, 'mom': {'release': '1.el7_0', 'buildtime': 1405326698L, 'version': '0.4.1'}}, 'numaNodeDistance': {'0': [10]}, 'cpuModel': 'Intel(R) Xeon(R) CPU E5645 @ 2.40GHz', 'liveMerge': 'false', 'hooks': {}, 'cpuSockets': '1', 'vmTypes': ['kvm'], 'selinux': {'mode': '1'}, 'kdumpStatus': 0, 'supportedProtocols': ['2.2', '2.3'], 'networks': {'rhevm': {'iface': 'rhevm', 'addr': '10.34.63.137', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'bridged': True, 'ipv6addrs': ['2620:52:0:223c:868f:69ff:fe67:1f0f/64', 'fe80::868f:69ff:fe67:1f0f/64'], 'gateway': '10.34.63.254', 'bootproto4': 'dhcp', 'netmask': '255.255.252.0', 'stp': 'off', 'ipv4addrs': ['10.34.63.137/22'], 'mtu': '1500', 'ipv6gateway': 'fe80:52:0:223c::3fe', 'ports': ['em1']}}, 'bridges': {'rhevm': {'addr': '10.34.63.137', 'cfg': {'DEFROUTE': 'yes', 'HOTPLUG': 'no', 'MTU': '1500', 'DELAY': '0', 'NM_CONTROLLED': 'no', 'BOOTPROTO': 'dhcp', 'STP': 'off', 'DEVICE': 'rhevm', 'TYPE': 'Bridge', 'ONBOOT': 'yes'}, 'ipv6addrs': ['2620:52:0:223c:868f:69ff:fe67:1f0f/64', 'fe80::868f:69ff:fe67:1f0f/64'], 'mtu': '1500', 'netmask': '255.255.252.0', 'stp': 'off', 'ipv4addrs': ['10.34.63.137/22'], 'ipv6gateway': 'fe80:52:0:223c::3fe', 'gateway': '10.34.63.254', 'opts': {'multicast_last_member_count': '2', 'hash_elasticity': '4', 'multicast_query_response_interval': '1000', 'group_fwd_mask': '0x0', 'multicast_snooping': '1', 'multicast_startup_query_interval': '3125', 'hello_timer': '114', 'multicast_querier_interval': '25500', 'max_age': '2000', 'hash_max': '512', 'stp_state': '0', 'topology_change_detected': '0', 'priority': '32768', 'multicast_membership_interval': '26000', 'root_path_cost': '0', 'root_port': '0', 'multicast_querier': '0', 'multicast_startup_query_count': '2', 'nf_call_iptables': '0', 'topology_change': '0', 'hello_time': '200', 'root_id': '8000.848f69671f0f', 'bridge_id': '8000.848f69671f0f', 'topology_change_timer': '0', 'ageing_time': '30000', 'nf_call_ip6tables': '0', 'gc_timer': '216', 'nf_call_arptables': '0', 'group_addr': '1:80:c2:0:0:0', 'multicast_last_member_interval': '100', 'multicast_query_interval': '12500', 'tcn_timer': '0', 'multicast_router': '1', 'vlan_filtering': '0', 'forward_delay': '0'}, 'ports': ['em1']}}, 'uuid': '4C4C4544-0052-4C10-8058-B5C04F43354A', 'lastClientIface': 'rhevm', 'nics': {'em1': {'addr': '', 'cfg': {'BRIDGE': 'rhevm', 'NM_CONTROLLED': 'no', 'MTU': '1500', 'HWADDR': '84:8f:69:67:1f:0f', 'DEVICE': 'em1', 'ONBOOT': 'yes'}, 'ipv6addrs': ['fe80::868f:69ff:fe67:1f0f/64'], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '84:8f:69:67:1f:0f', 'speed': 1000}, 'em2': {'addr': '', 'cfg': {'PEERROUTES': 'yes', 'IPV6INIT': 'yes', 'NAME': 'em2', 'IPV6_PEERDNS': 'yes', 'DEFROUTE': 'yes', 'UUID': '6d707312-7e57-4ec0-ba12-7cb025979f88', 'PEERDNS': 'yes', 'IPV4_FAILURE_FATAL': 'no', 'IPV6_AUTOCONF': 'yes', 'BOOTPROTO': 'dhcp', 'IPV6_DEFROUTE': 'yes', 'HWADDR': '84:8F:69:67:1F:11', 'IPV6_FAILURE_FATAL': 'no', 'TYPE': 'Ethernet', 'ONBOOT': 'no', 'IPV6_PEERROUTES': 'yes'}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '84:8f:69:67:1f:11', 'speed': 0}}, 'software_revision': '3', 'clusterLevels': ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'], 'cpuFlags': u'fpu,vme,de,pse,tsc,msr,pae,mce,cx8,apic,sep,mtrr,pge,mca,cmov,pat,pse36,clflush,dts,acpi,mmx,fxsr,sse,sse2,ss,ht,tm,pbe,syscall,nx,pdpe1gb,rdtscp,lm,constant_tsc,arch_perfmon,pebs,bts,rep_good,nopl,xtopology,nonstop_tsc,aperfmperf,pni,pclmulqdq,dtes64,monitor,ds_cpl,vmx,smx,est,tm2,ssse3,cx16,xtpr,pdcm,pcid,dca,sse4_1,sse4_2,popcnt,aes,lahf_lm,ida,arat,dtherm,tpr_shadow,vnmi,flexpriority,ept,vpid,model_Nehalem,model_Conroe,model_coreduo,model_core2duo,model_Penryn,model_Westmere,model_n270', 'ISCSIInitiatorName': 'iqn.1994-05.com.redhat:3bd6acb1d4f2', 'netConfigDirty': 'False', 'supportedENGINEs': ['3.0', '3.1', '3.2', '3.3', '3.4', '3.5'], 'autoNumaBalancing': 0, 'reservedMem': '321', 'bondings': {'bond0': {'addr': '', 'cfg': {}, 'ipv6addrs': [], 'mtu': '1500', 'netmask': '', 'ipv4addrs': [], 'hwaddr': '76:ee:22:47:c3:e4', 'slaves': [], 'opts': {}}}, 'software_version': '4.16', 'memSize': '19903', 'cpuSpeed': '2393.935', 'numaNodes': {u'0': {'totalMemory': '19903', 'cpus': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]}}, 'version_name': 'Snow Man', 'vlans': {}, 'cpuCores': '6', 'kvmEnabled': 'true', 'guestOverhead': '65', 'supportedRHEVMs': ['3.0'], 'cpuThreads': '12', 'emulatedMachines': [u'pc', u'q35', u'rhel6.0.0', u'rhel6.1.0', u'rhel6.2.0', u'rhel6.3.0', u'rhel6.4.0', u'rhel6.5.0', u'none'], 'rngSources': ['random'], 'operatingSystem': {'release': '1.el7', 'version': '7.0', 'name': 'RHEL'}} >Thread-71676::DEBUG::2014-09-25 18:05:32,356::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:32,409::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71677::DEBUG::2014-09-25 18:05:32,410::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getHardwareInfo' in bridge with {} >JsonRpcServer::DEBUG::2014-09-25 18:05:32,410::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71677::DEBUG::2014-09-25 18:05:32,414::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getHardwareInfo' in bridge with {'systemProductName': 'PowerEdge M610', 'systemUUID': '4C4C4544-0052-4C10-8058-B5C04F43354A', 'systemSerialNumber': '5RLXC5J', 'systemManufacturer': 'Dell Inc.'} >Thread-71677::DEBUG::2014-09-25 18:05:32,414::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:32,471::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:32,472::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71678::DEBUG::2014-09-25 18:05:32,474::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:05:33,280::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:33,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:33,280::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:33,280::__init__::232::IOProcess::(_processLogs) (110413) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:33,283::__init__::232::IOProcess::(_processLogs) (110413) Queuing response >Thread-79::DEBUG::2014-09-25 18:05:33,285::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:05:33,301::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00532047 s, 106 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:33,302::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:33,302::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:33,303::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:33,303::__init__::232::IOProcess::(_processLogs) (110414) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:33,306::__init__::232::IOProcess::(_processLogs) (110414) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:33,307::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:33,308::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:33,308::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:33,309::__init__::232::IOProcess::(_processLogs) (110415) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:33,309::__init__::232::IOProcess::(_processLogs) (110415) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:33,310::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:33,310::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:33,311::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:33,311::__init__::232::IOProcess::(_processLogs) (110416) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:33,311::__init__::232::IOProcess::(_processLogs) (110416) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:33,312::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:33,313::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:33,313::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:33,313::__init__::232::IOProcess::(_processLogs) (110417) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:33,314::__init__::232::IOProcess::(_processLogs) (110417) Queuing response >Dummy-148::DEBUG::2014-09-25 18:05:33,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:33,900::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200579 s, 51.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:35,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:35,950::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207522 s, 49.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:36,046::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:36,047::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71679::DEBUG::2014-09-25 18:05:36,050::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:05:36,965::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:36,966::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:36,966::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:36,966::__init__::232::IOProcess::(_processLogs) (110418) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:36,967::__init__::232::IOProcess::(_processLogs) (110418) Queuing response >Thread-81::DEBUG::2014-09-25 18:05:36,968::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:05:36,976::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000439876 s, 843 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:36,978::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:36,978::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:36,979::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:36,980::__init__::232::IOProcess::(_processLogs) (110419) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:36,980::__init__::232::IOProcess::(_processLogs) (110419) Queuing response >Dummy-148::DEBUG::2014-09-25 18:05:37,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:37,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146367 s, 70.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:39,075::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:39,076::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71680::DEBUG::2014-09-25 18:05:39,078::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:39,176::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71681::DEBUG::2014-09-25 18:05:39,177::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:05:39,177::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71681::DEBUG::2014-09-25 18:05:39,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`56ca83af-ab6c-4cb0-9589-17ee4b3654a8`::moving from state init -> state preparing >Thread-71681::INFO::2014-09-25 18:05:39,180::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71681::INFO::2014-09-25 18:05:39,180::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71681::DEBUG::2014-09-25 18:05:39,181::task::1191::Storage.TaskManager.Task::(prepare) Task=`56ca83af-ab6c-4cb0-9589-17ee4b3654a8`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71681::DEBUG::2014-09-25 18:05:39,181::task::595::Storage.TaskManager.Task::(_updateState) Task=`56ca83af-ab6c-4cb0-9589-17ee4b3654a8`::moving from state preparing -> state finished >Thread-71681::DEBUG::2014-09-25 18:05:39,181::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71681::DEBUG::2014-09-25 18:05:39,181::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71681::DEBUG::2014-09-25 18:05:39,182::task::993::Storage.TaskManager.Task::(_decref) Task=`56ca83af-ab6c-4cb0-9589-17ee4b3654a8`::ref 0 aborting False >Thread-71681::DEBUG::2014-09-25 18:05:39,182::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71681::DEBUG::2014-09-25 18:05:39,182::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Detector thread::DEBUG::2014-09-25 18:05:39,279::protocoldetector::160::vds.MultiProtocolAcceptor::(_add_connection) Adding connection from 10.34.63.74:54608 >Detector thread::DEBUG::2014-09-25 18:05:39,280::protocoldetector::171::vds.MultiProtocolAcceptor::(_remove_connection) Connection removed from 10.34.63.74:54608 >Detector thread::DEBUG::2014-09-25 18:05:39,280::protocoldetector::197::vds.MultiProtocolAcceptor::(_handle_connection_read) Detected protocol stomp from 10.34.63.74:54608 >Detector thread::DEBUG::2014-09-25 18:05:39,281::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='CONNECT'> >Detector thread::INFO::2014-09-25 18:05:39,281::stompReactor::66::Broker.StompAdapter::(_cmd_connect) Processing CONNECT request >Detector thread::DEBUG::2014-09-25 18:05:39,281::stompReactor::338::protocoldetector.StompDetector::(handleSocket) Stomp detected from ('10.34.63.74', 54608) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:39,283::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SUBSCRIBE'> >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:39,283::stompReactor::89::Broker.StompAdapter::(_cmd_subscribe) Subscribe command ignored >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:39,323::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71682::DEBUG::2014-09-25 18:05:39,324::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:05:39,324::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71682::DEBUG::2014-09-25 18:05:39,326::task::595::Storage.TaskManager.Task::(_updateState) Task=`e00978b0-f863-433e-b0dc-5433d11431a6`::moving from state init -> state preparing >Thread-71682::INFO::2014-09-25 18:05:39,326::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71682::DEBUG::2014-09-25 18:05:39,327::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2ea538c2-4ca5-430f-b6fb-5292e1d568eb`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71682::DEBUG::2014-09-25 18:05:39,327::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71682::DEBUG::2014-09-25 18:05:39,328::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71682::DEBUG::2014-09-25 18:05:39,328::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`2ea538c2-4ca5-430f-b6fb-5292e1d568eb`::Granted request >Thread-71682::DEBUG::2014-09-25 18:05:39,328::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`e00978b0-f863-433e-b0dc-5433d11431a6`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71682::DEBUG::2014-09-25 18:05:39,329::task::993::Storage.TaskManager.Task::(_decref) Task=`e00978b0-f863-433e-b0dc-5433d11431a6`::ref 1 aborting False >Thread-71682::INFO::2014-09-25 18:05:39,333::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71682::DEBUG::2014-09-25 18:05:39,333::task::1191::Storage.TaskManager.Task::(prepare) Task=`e00978b0-f863-433e-b0dc-5433d11431a6`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71682::DEBUG::2014-09-25 18:05:39,334::task::595::Storage.TaskManager.Task::(_updateState) Task=`e00978b0-f863-433e-b0dc-5433d11431a6`::moving from state preparing -> state finished >Thread-71682::DEBUG::2014-09-25 18:05:39,334::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71682::DEBUG::2014-09-25 18:05:39,334::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71682::DEBUG::2014-09-25 18:05:39,334::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71682::DEBUG::2014-09-25 18:05:39,335::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71682::DEBUG::2014-09-25 18:05:39,335::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71682::DEBUG::2014-09-25 18:05:39,335::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71682::DEBUG::2014-09-25 18:05:39,335::task::993::Storage.TaskManager.Task::(_decref) Task=`e00978b0-f863-433e-b0dc-5433d11431a6`::ref 0 aborting False >Thread-71682::DEBUG::2014-09-25 18:05:39,336::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71682::DEBUG::2014-09-25 18:05:39,336::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:39,400::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71683::DEBUG::2014-09-25 18:05:39,401::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getAllTasksInfo' in bridge with {} >JsonRpcServer::DEBUG::2014-09-25 18:05:39,401::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71683::DEBUG::2014-09-25 18:05:39,403::task::595::Storage.TaskManager.Task::(_updateState) Task=`2ae8fc00-1cff-4bfd-a683-84fed832a9c8`::moving from state init -> state preparing >Thread-71683::INFO::2014-09-25 18:05:39,403::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksInfo(spUUID=None, options=None) >Thread-71683::DEBUG::2014-09-25 18:05:39,404::taskManager::191::Storage.TaskManager::(getAllTasksInfo) Entry. >Thread-71683::DEBUG::2014-09-25 18:05:39,404::taskManager::202::Storage.TaskManager::(getAllTasksInfo) Return. Response: {} >Thread-71683::INFO::2014-09-25 18:05:39,404::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksInfo, Return response: {'allTasksInfo': {}} >Thread-71683::DEBUG::2014-09-25 18:05:39,404::task::1191::Storage.TaskManager.Task::(prepare) Task=`2ae8fc00-1cff-4bfd-a683-84fed832a9c8`::finished: {'allTasksInfo': {}} >Thread-71683::DEBUG::2014-09-25 18:05:39,404::task::595::Storage.TaskManager.Task::(_updateState) Task=`2ae8fc00-1cff-4bfd-a683-84fed832a9c8`::moving from state preparing -> state finished >Thread-71683::DEBUG::2014-09-25 18:05:39,405::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71683::DEBUG::2014-09-25 18:05:39,405::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71683::DEBUG::2014-09-25 18:05:39,405::task::993::Storage.TaskManager.Task::(_decref) Task=`2ae8fc00-1cff-4bfd-a683-84fed832a9c8`::ref 0 aborting False >Thread-71683::DEBUG::2014-09-25 18:05:39,405::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getAllTasksInfo' in bridge with {} >Thread-71683::DEBUG::2014-09-25 18:05:39,406::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:05:40,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:40,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178987 s, 57.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:05:42,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:42,091::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014062 s, 72.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:42,104::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:42,107::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71684::DEBUG::2014-09-25 18:05:42,112::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:05:43,313::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:43,314::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:43,314::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:43,314::__init__::232::IOProcess::(_processLogs) (110420) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:43,315::__init__::232::IOProcess::(_processLogs) (110420) Queuing response >Thread-79::DEBUG::2014-09-25 18:05:43,317::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:05:43,325::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000395249 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:43,327::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:43,327::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:43,327::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:43,327::__init__::232::IOProcess::(_processLogs) (110421) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:43,328::__init__::232::IOProcess::(_processLogs) (110421) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:43,330::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:43,330::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:43,330::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:43,330::__init__::232::IOProcess::(_processLogs) (110422) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:43,331::__init__::232::IOProcess::(_processLogs) (110422) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:43,332::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:43,332::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:43,332::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:43,333::__init__::232::IOProcess::(_processLogs) (110423) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:43,333::__init__::232::IOProcess::(_processLogs) (110423) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:43,334::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:43,334::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:43,334::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:43,335::__init__::232::IOProcess::(_processLogs) (110424) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:43,335::__init__::232::IOProcess::(_processLogs) (110424) Queuing response >Dummy-148::DEBUG::2014-09-25 18:05:44,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:44,143::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133909 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:45,143::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:45,145::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71685::DEBUG::2014-09-25 18:05:45,147::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:05:46,164::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:46,207::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0298397 s, 34.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:46,982::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:46,983::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:46,985::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:46,986::__init__::232::IOProcess::(_processLogs) (110425) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:46,988::__init__::232::IOProcess::(_processLogs) (110425) Queuing response >Thread-81::DEBUG::2014-09-25 18:05:46,990::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:05:47,016::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00810473 s, 45.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:47,017::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:47,018::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:47,018::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:47,018::__init__::232::IOProcess::(_processLogs) (110426) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:47,023::__init__::232::IOProcess::(_processLogs) (110426) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:48,198::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:48,200::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71686::DEBUG::2014-09-25 18:05:48,202::task::595::Storage.TaskManager.Task::(_updateState) Task=`b563686a-0c12-4fc2-bd13-ab3da95cd4bb`::moving from state init -> state preparing >Thread-71686::INFO::2014-09-25 18:05:48,202::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71686::INFO::2014-09-25 18:05:48,203::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000395249', 'lastCheck': '4.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '60.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00810473', 'lastCheck': '1.2', 'valid': True}} >Thread-71686::DEBUG::2014-09-25 18:05:48,203::task::1191::Storage.TaskManager.Task::(prepare) Task=`b563686a-0c12-4fc2-bd13-ab3da95cd4bb`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000395249', 'lastCheck': '4.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 200, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '60.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00810473', 'lastCheck': '1.2', 'valid': True}} >Thread-71686::DEBUG::2014-09-25 18:05:48,204::task::595::Storage.TaskManager.Task::(_updateState) Task=`b563686a-0c12-4fc2-bd13-ab3da95cd4bb`::moving from state preparing -> state finished >Thread-71686::DEBUG::2014-09-25 18:05:48,204::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71686::DEBUG::2014-09-25 18:05:48,204::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71686::DEBUG::2014-09-25 18:05:48,204::task::993::Storage.TaskManager.Task::(_decref) Task=`b563686a-0c12-4fc2-bd13-ab3da95cd4bb`::ref 0 aborting False >Thread-71686::DEBUG::2014-09-25 18:05:48,207::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:48,234::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:48,235::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71687::DEBUG::2014-09-25 18:05:48,237::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:05:48,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:48,412::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162732 s, 62.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:49,398::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:49,400::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71688::DEBUG::2014-09-25 18:05:49,400::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71688::DEBUG::2014-09-25 18:05:49,402::task::595::Storage.TaskManager.Task::(_updateState) Task=`56046ca4-c956-4d35-8b13-1cabcd219469`::moving from state init -> state preparing >Thread-71688::INFO::2014-09-25 18:05:49,402::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71688::INFO::2014-09-25 18:05:49,402::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71688::DEBUG::2014-09-25 18:05:49,403::task::1191::Storage.TaskManager.Task::(prepare) Task=`56046ca4-c956-4d35-8b13-1cabcd219469`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71688::DEBUG::2014-09-25 18:05:49,403::task::595::Storage.TaskManager.Task::(_updateState) Task=`56046ca4-c956-4d35-8b13-1cabcd219469`::moving from state preparing -> state finished >Thread-71688::DEBUG::2014-09-25 18:05:49,403::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71688::DEBUG::2014-09-25 18:05:49,403::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71688::DEBUG::2014-09-25 18:05:49,404::task::993::Storage.TaskManager.Task::(_decref) Task=`56046ca4-c956-4d35-8b13-1cabcd219469`::ref 0 aborting False >Thread-71688::DEBUG::2014-09-25 18:05:49,404::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71688::DEBUG::2014-09-25 18:05:49,404::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:49,422::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:49,423::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71689::DEBUG::2014-09-25 18:05:49,423::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71689::DEBUG::2014-09-25 18:05:49,425::task::595::Storage.TaskManager.Task::(_updateState) Task=`496ede09-01c7-4c48-8332-e932c9ac4c2f`::moving from state init -> state preparing >Thread-71689::INFO::2014-09-25 18:05:49,425::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71689::DEBUG::2014-09-25 18:05:49,426::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`329c51f6-7822-4669-93ae-274c928919cf`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71689::DEBUG::2014-09-25 18:05:49,426::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71689::DEBUG::2014-09-25 18:05:49,427::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71689::DEBUG::2014-09-25 18:05:49,427::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`329c51f6-7822-4669-93ae-274c928919cf`::Granted request >Thread-71689::DEBUG::2014-09-25 18:05:49,427::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`496ede09-01c7-4c48-8332-e932c9ac4c2f`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71689::DEBUG::2014-09-25 18:05:49,428::task::993::Storage.TaskManager.Task::(_decref) Task=`496ede09-01c7-4c48-8332-e932c9ac4c2f`::ref 1 aborting False >Thread-71689::INFO::2014-09-25 18:05:49,432::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71689::DEBUG::2014-09-25 18:05:49,432::task::1191::Storage.TaskManager.Task::(prepare) Task=`496ede09-01c7-4c48-8332-e932c9ac4c2f`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71689::DEBUG::2014-09-25 18:05:49,432::task::595::Storage.TaskManager.Task::(_updateState) Task=`496ede09-01c7-4c48-8332-e932c9ac4c2f`::moving from state preparing -> state finished >Thread-71689::DEBUG::2014-09-25 18:05:49,433::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71689::DEBUG::2014-09-25 18:05:49,433::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71689::DEBUG::2014-09-25 18:05:49,433::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71689::DEBUG::2014-09-25 18:05:49,433::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71689::DEBUG::2014-09-25 18:05:49,434::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71689::DEBUG::2014-09-25 18:05:49,434::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71689::DEBUG::2014-09-25 18:05:49,434::task::993::Storage.TaskManager.Task::(_decref) Task=`496ede09-01c7-4c48-8332-e932c9ac4c2f`::ref 0 aborting False >Thread-71689::DEBUG::2014-09-25 18:05:49,435::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71689::DEBUG::2014-09-25 18:05:49,435::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:05:50,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:50,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0248533 s, 41.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:51,582::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:51,584::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71690::DEBUG::2014-09-25 18:05:51,586::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:05:52,487::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:52,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0187987 s, 54.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:53,336::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:53,337::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:53,337::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:53,337::__init__::232::IOProcess::(_processLogs) (110427) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:53,338::__init__::232::IOProcess::(_processLogs) (110427) Queuing response >Thread-79::DEBUG::2014-09-25 18:05:53,339::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:05:53,348::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000358091 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:53,349::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:53,349::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:53,350::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:53,350::__init__::232::IOProcess::(_processLogs) (110428) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:53,350::__init__::232::IOProcess::(_processLogs) (110428) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:53,352::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:53,352::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:53,353::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:53,353::__init__::232::IOProcess::(_processLogs) (110429) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:53,353::__init__::232::IOProcess::(_processLogs) (110429) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:53,354::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:53,354::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:53,354::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:53,355::__init__::232::IOProcess::(_processLogs) (110430) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:53,356::__init__::232::IOProcess::(_processLogs) (110430) Queuing response >Thread-29::DEBUG::2014-09-25 18:05:53,356::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:53,356::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:53,357::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:53,358::__init__::232::IOProcess::(_processLogs) (110431) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:05:53,358::__init__::232::IOProcess::(_processLogs) (110431) Queuing response >Dummy-148::DEBUG::2014-09-25 18:05:54,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:54,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174752 s, 58.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:54,603::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:54,604::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71691::DEBUG::2014-09-25 18:05:54,606::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:05:56,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:56,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173249 s, 59.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:57,025::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:57,025::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:57,026::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:57,026::__init__::232::IOProcess::(_processLogs) (110432) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:57,026::__init__::232::IOProcess::(_processLogs) (110432) Queuing response >Thread-81::DEBUG::2014-09-25 18:05:57,027::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:05:57,036::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000454429 s, 816 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:05:57,037::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:57,037::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:57,038::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:57,039::__init__::232::IOProcess::(_processLogs) (110433) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:05:57,039::__init__::232::IOProcess::(_processLogs) (110433) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:57,636::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:57,638::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71692::DEBUG::2014-09-25 18:05:57,640::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-71673::WARNING::2014-09-25 18:05:57,987::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:05:57,988::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:05:57,989::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Dummy-148::DEBUG::2014-09-25 18:05:58,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:05:58,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148959 s, 68.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:58,741::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:58,742::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71693::DEBUG::2014-09-25 18:05:58,743::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.connectStorageServer' in bridge with {u'connectionParams': [{u'id': u'6d77ae90-0a92-4f76-9980-cb7eceb7df64', u'connection': u'10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': u'', u'port': u''}, {u'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8', u'connection': u'pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', u'iqn': u'', u'user': u'', u'tpgt': u'', u'password': u'', u'port': u''}, {u'id': u'd7af8c51-78e6-44db-88d2-f765dc508251', u'connection': u'10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': u'', u'port': u''}], u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad', u'domainType': 1} >Thread-71693::DEBUG::2014-09-25 18:05:58,745::task::595::Storage.TaskManager.Task::(_updateState) Task=`38036347-d013-4b56-8e1b-92c4e200428f`::moving from state init -> state preparing >Thread-71693::INFO::2014-09-25 18:05:58,745::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID=u'00000002-0002-0002-0002-0000000001ad', conList=[{u'connection': u'10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'6d77ae90-0a92-4f76-9980-cb7eceb7df64', u'port': u''}, {u'connection': u'pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', u'iqn': u'', u'user': u'', u'tpgt': u'', u'password': '******', u'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8', u'port': u''}, {u'connection': u'10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', u'iqn': u'', u'user': u'', u'tpgt': u'1', u'password': '******', u'id': u'd7af8c51-78e6-44db-88d2-f765dc508251', u'port': u''}], options=None) >Thread-71693::DEBUG::2014-09-25 18:05:58,749::hsm::2372::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export >Thread-29::DEBUG::2014-09-25 18:05:58,750::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:58,750::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:58,750::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:58,750::__init__::232::IOProcess::(_processLogs) (110434) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:05:58,751::__init__::232::IOProcess::(_processLogs) (110434) Queuing response >Thread-71693::DEBUG::2014-09-25 18:05:58,752::hsm::2396::Storage.HSM::(__prefetchDomains) Found SD uuids: (u'3573f707-d0c4-4dc9-b09f-dca9a3506a4f',) >Thread-71693::DEBUG::2014-09-25 18:05:58,752::hsm::2452::Storage.HSM::(connectStorageServer) knownSDs: {3573f707-d0c4-4dc9-b09f-dca9a3506a4f: storage.nfsSD.findDomain, fc587b6a-bac3-4668-99de-7f6fd6acce7f: storage.nfsSD.findDomain, eb633636-3019-47a9-9f5a-fc81f8dfd7dc: storage.nfsSD.findDomain} >Thread-71693::DEBUG::2014-09-25 18:05:58,756::hsm::2372::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Thread-29::DEBUG::2014-09-25 18:05:58,756::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:05:58,756::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:05:58,757::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:05:58,757::__init__::232::IOProcess::(_processLogs) (110435) Got request for method 'glob' >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:59,488::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71694::DEBUG::2014-09-25 18:05:59,490::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:05:59,489::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71694::DEBUG::2014-09-25 18:05:59,492::task::595::Storage.TaskManager.Task::(_updateState) Task=`7e1fc8aa-738d-42ef-a83f-9c4f5f08e0c3`::moving from state init -> state preparing >Thread-71694::INFO::2014-09-25 18:05:59,492::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71694::INFO::2014-09-25 18:05:59,492::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71694::DEBUG::2014-09-25 18:05:59,493::task::1191::Storage.TaskManager.Task::(prepare) Task=`7e1fc8aa-738d-42ef-a83f-9c4f5f08e0c3`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71694::DEBUG::2014-09-25 18:05:59,493::task::595::Storage.TaskManager.Task::(_updateState) Task=`7e1fc8aa-738d-42ef-a83f-9c4f5f08e0c3`::moving from state preparing -> state finished >Thread-71694::DEBUG::2014-09-25 18:05:59,493::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71694::DEBUG::2014-09-25 18:05:59,493::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71694::DEBUG::2014-09-25 18:05:59,494::task::993::Storage.TaskManager.Task::(_decref) Task=`7e1fc8aa-738d-42ef-a83f-9c4f5f08e0c3`::ref 0 aborting False >Thread-71694::DEBUG::2014-09-25 18:05:59,494::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71694::DEBUG::2014-09-25 18:05:59,494::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:05:59,531::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:05:59,532::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71695::DEBUG::2014-09-25 18:05:59,532::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71695::DEBUG::2014-09-25 18:05:59,534::task::595::Storage.TaskManager.Task::(_updateState) Task=`04223a5a-01a0-4271-8288-1bcb822fe839`::moving from state init -> state preparing >Thread-71695::INFO::2014-09-25 18:05:59,534::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71695::DEBUG::2014-09-25 18:05:59,535::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f912afdd-829d-484a-84cd-ac2bd0ecf938`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71695::DEBUG::2014-09-25 18:05:59,535::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71695::DEBUG::2014-09-25 18:05:59,536::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71695::DEBUG::2014-09-25 18:05:59,536::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`f912afdd-829d-484a-84cd-ac2bd0ecf938`::Granted request >Thread-71695::DEBUG::2014-09-25 18:05:59,536::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`04223a5a-01a0-4271-8288-1bcb822fe839`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71695::DEBUG::2014-09-25 18:05:59,537::task::993::Storage.TaskManager.Task::(_decref) Task=`04223a5a-01a0-4271-8288-1bcb822fe839`::ref 1 aborting False >Thread-71695::INFO::2014-09-25 18:05:59,541::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71695::DEBUG::2014-09-25 18:05:59,541::task::1191::Storage.TaskManager.Task::(prepare) Task=`04223a5a-01a0-4271-8288-1bcb822fe839`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71695::DEBUG::2014-09-25 18:05:59,541::task::595::Storage.TaskManager.Task::(_updateState) Task=`04223a5a-01a0-4271-8288-1bcb822fe839`::moving from state preparing -> state finished >Thread-71695::DEBUG::2014-09-25 18:05:59,542::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71695::DEBUG::2014-09-25 18:05:59,542::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71695::DEBUG::2014-09-25 18:05:59,542::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71695::DEBUG::2014-09-25 18:05:59,543::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71695::DEBUG::2014-09-25 18:05:59,543::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71695::DEBUG::2014-09-25 18:05:59,543::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71695::DEBUG::2014-09-25 18:05:59,544::task::993::Storage.TaskManager.Task::(_decref) Task=`04223a5a-01a0-4271-8288-1bcb822fe839`::ref 0 aborting False >Thread-71695::DEBUG::2014-09-25 18:05:59,544::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71695::DEBUG::2014-09-25 18:05:59,544::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:00,664::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:00,665::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71696::DEBUG::2014-09-25 18:06:00,667::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:00,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:00,714::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214305 s, 47.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:06:02,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:02,981::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.237626 s, 4.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:03,358::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:03,359::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:03,359::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:03,359::__init__::232::IOProcess::(_processLogs) (110436) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:03,368::__init__::232::IOProcess::(_processLogs) (110436) Queuing response >Thread-79::DEBUG::2014-09-25 18:06:03,368::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:06:03,397::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00940891 s, 59.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:03,398::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:03,399::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:03,399::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:03,399::__init__::232::IOProcess::(_processLogs) (110437) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:03,428::__init__::232::IOProcess::(_processLogs) (110437) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:03,430::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:03,430::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:03,430::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:03,431::__init__::232::IOProcess::(_processLogs) (110438) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:03,431::__init__::232::IOProcess::(_processLogs) (110438) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:03,432::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:03,432::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:03,433::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:03,433::__init__::232::IOProcess::(_processLogs) (110439) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:03,434::__init__::232::IOProcess::(_processLogs) (110439) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:03,434::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:03,434::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:03,435::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:03,436::__init__::232::IOProcess::(_processLogs) (110440) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:03,436::__init__::232::IOProcess::(_processLogs) (110440) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:03,688::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:03,689::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71697::DEBUG::2014-09-25 18:06:03,692::task::595::Storage.TaskManager.Task::(_updateState) Task=`e2bce206-cfdd-4ff4-8811-3655a93e51c7`::moving from state init -> state preparing >Thread-71697::INFO::2014-09-25 18:06:03,692::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71697::INFO::2014-09-25 18:06:03,693::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00940891', 'lastCheck': '0.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000454429', 'lastCheck': '6.7', 'valid': True}} >Thread-71697::DEBUG::2014-09-25 18:06:03,693::task::1191::Storage.TaskManager.Task::(prepare) Task=`e2bce206-cfdd-4ff4-8811-3655a93e51c7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00940891', 'lastCheck': '0.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '5.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000454429', 'lastCheck': '6.7', 'valid': True}} >Thread-71697::DEBUG::2014-09-25 18:06:03,693::task::595::Storage.TaskManager.Task::(_updateState) Task=`e2bce206-cfdd-4ff4-8811-3655a93e51c7`::moving from state preparing -> state finished >Thread-71697::DEBUG::2014-09-25 18:06:03,694::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71697::DEBUG::2014-09-25 18:06:03,694::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71697::DEBUG::2014-09-25 18:06:03,694::task::993::Storage.TaskManager.Task::(_decref) Task=`e2bce206-cfdd-4ff4-8811-3655a93e51c7`::ref 0 aborting False >Thread-71697::DEBUG::2014-09-25 18:06:03,696::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:03,716::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:03,718::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71698::DEBUG::2014-09-25 18:06:03,720::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:05,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:05,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131301 s, 78.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:06,786::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:06,788::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71699::DEBUG::2014-09-25 18:06:06,790::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:06:07,044::__init__::232::IOProcess::(_processLogs) Receiving request... >Dummy-148::DEBUG::2014-09-25 18:06:07,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 18:06:07,045::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,054::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,054::__init__::232::IOProcess::(_processLogs) (110441) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:07,055::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,055::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,055::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,055::__init__::232::IOProcess::(_processLogs) (110442) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:07,056::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,056::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,056::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,056::__init__::232::IOProcess::(_processLogs) (110443) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:07,057::__init__::232::IOProcess::(_processLogs) (110441) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:07,061::__init__::232::IOProcess::(_processLogs) (110443) Queuing response >Thread-81::DEBUG::2014-09-25 18:06:07,066::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 18:06:07,068::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,068::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,068::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,069::__init__::232::IOProcess::(_processLogs) (110444) Got request for method 'touch' >Dummy-148::DEBUG::2014-09-25 18:06:07,077::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194038 s, 52.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:07,107::__init__::232::IOProcess::(_processLogs) (110444) Queuing response >Thread-81::DEBUG::2014-09-25 18:06:07,108::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:06:07,108::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,109::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,109::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,109::__init__::232::IOProcess::(_processLogs) (110445) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 18:06:07,119::__init__::232::IOProcess::(_processLogs) (110445) Queuing response >Thread-81::DEBUG::2014-09-25 18:06:07,119::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 18:06:07,120::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,121::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,121::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,122::__init__::232::IOProcess::(_processLogs) (110446) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:07,122::__init__::232::IOProcess::(_processLogs) (110446) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:07,122::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,123::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,123::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,123::__init__::232::IOProcess::(_processLogs) (110447) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:07,128::__init__::232::IOProcess::(_processLogs) (110447) Queuing response >Thread-81::DEBUG::2014-09-25 18:06:07,129::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 18:06:07,129::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 18:06:07,129::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 18:06:07,130::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,130::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,130::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,131::__init__::232::IOProcess::(_processLogs) (110448) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:07,134::__init__::232::IOProcess::(_processLogs) (110448) Queuing response >Thread-81::DEBUG::2014-09-25 18:06:07,135::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:06:07,152::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00401941 s, 92.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:07,153::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,153::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,154::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,154::__init__::232::IOProcess::(_processLogs) (110449) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:07,155::__init__::232::IOProcess::(_processLogs) (110449) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:07,994::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,995::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,995::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,995::__init__::232::IOProcess::(_processLogs) (110450) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:07,996::__init__::232::IOProcess::(_processLogs) (110450) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:07,996::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,996::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:07,996::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,997::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:07,997::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,997::__init__::232::IOProcess::(_processLogs) (110451) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:07,997::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:07,998::__init__::232::IOProcess::(_processLogs) (110452) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:07,998::__init__::232::IOProcess::(_processLogs) (110452) Queuing response >Dummy-148::DEBUG::2014-09-25 18:06:09,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:09,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.265075 s, 3.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:09,576::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:09,578::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71706::DEBUG::2014-09-25 18:06:09,578::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71706::DEBUG::2014-09-25 18:06:09,580::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f950119-2ecb-4d32-9cac-ac1c0e51e5a6`::moving from state init -> state preparing >Thread-71706::INFO::2014-09-25 18:06:09,580::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71706::INFO::2014-09-25 18:06:09,581::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71706::DEBUG::2014-09-25 18:06:09,581::task::1191::Storage.TaskManager.Task::(prepare) Task=`2f950119-2ecb-4d32-9cac-ac1c0e51e5a6`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71706::DEBUG::2014-09-25 18:06:09,581::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f950119-2ecb-4d32-9cac-ac1c0e51e5a6`::moving from state preparing -> state finished >Thread-71706::DEBUG::2014-09-25 18:06:09,581::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71706::DEBUG::2014-09-25 18:06:09,582::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71706::DEBUG::2014-09-25 18:06:09,582::task::993::Storage.TaskManager.Task::(_decref) Task=`2f950119-2ecb-4d32-9cac-ac1c0e51e5a6`::ref 0 aborting False >Thread-71706::DEBUG::2014-09-25 18:06:09,582::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71706::DEBUG::2014-09-25 18:06:09,583::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:09,674::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:09,675::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71707::DEBUG::2014-09-25 18:06:09,675::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71707::DEBUG::2014-09-25 18:06:09,677::task::595::Storage.TaskManager.Task::(_updateState) Task=`7640d08b-ca23-40c0-ba21-3ac5080563a8`::moving from state init -> state preparing >Thread-71707::INFO::2014-09-25 18:06:09,677::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71707::DEBUG::2014-09-25 18:06:09,678::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b49b632a-f07a-4614-88a7-2e28e85aca2b`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71707::DEBUG::2014-09-25 18:06:09,678::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71707::DEBUG::2014-09-25 18:06:09,678::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71707::DEBUG::2014-09-25 18:06:09,679::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`b49b632a-f07a-4614-88a7-2e28e85aca2b`::Granted request >Thread-71707::DEBUG::2014-09-25 18:06:09,679::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`7640d08b-ca23-40c0-ba21-3ac5080563a8`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71707::DEBUG::2014-09-25 18:06:09,679::task::993::Storage.TaskManager.Task::(_decref) Task=`7640d08b-ca23-40c0-ba21-3ac5080563a8`::ref 1 aborting False >Thread-71707::INFO::2014-09-25 18:06:09,684::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71707::DEBUG::2014-09-25 18:06:09,684::task::1191::Storage.TaskManager.Task::(prepare) Task=`7640d08b-ca23-40c0-ba21-3ac5080563a8`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71707::DEBUG::2014-09-25 18:06:09,684::task::595::Storage.TaskManager.Task::(_updateState) Task=`7640d08b-ca23-40c0-ba21-3ac5080563a8`::moving from state preparing -> state finished >Thread-71707::DEBUG::2014-09-25 18:06:09,685::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71707::DEBUG::2014-09-25 18:06:09,685::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71707::DEBUG::2014-09-25 18:06:09,685::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71707::DEBUG::2014-09-25 18:06:09,685::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71707::DEBUG::2014-09-25 18:06:09,686::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71707::DEBUG::2014-09-25 18:06:09,686::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71707::DEBUG::2014-09-25 18:06:09,686::task::993::Storage.TaskManager.Task::(_decref) Task=`7640d08b-ca23-40c0-ba21-3ac5080563a8`::ref 0 aborting False >Thread-71707::DEBUG::2014-09-25 18:06:09,687::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71707::DEBUG::2014-09-25 18:06:09,687::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:09,809::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:09,810::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71708::DEBUG::2014-09-25 18:06:09,812::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:11,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:11,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0239869 s, 42.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:12,830::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:12,831::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71709::DEBUG::2014-09-25 18:06:12,833::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:06:13,437::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:13,437::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:13,437::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:13,437::__init__::232::IOProcess::(_processLogs) (110453) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:13,439::__init__::232::IOProcess::(_processLogs) (110453) Queuing response >Thread-79::DEBUG::2014-09-25 18:06:13,440::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:13,448::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-79::DEBUG::2014-09-25 18:06:13,464::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00903574 s, 62.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:13,466::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:13,466::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:13,466::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:13,467::__init__::232::IOProcess::(_processLogs) (110454) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:13,470::__init__::232::IOProcess::(_processLogs) (110454) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:13,472::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:13,472::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:13,473::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:13,473::__init__::232::IOProcess::(_processLogs) (110455) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:13,473::__init__::232::IOProcess::(_processLogs) (110455) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:13,474::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:13,474::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:13,475::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:13,475::__init__::232::IOProcess::(_processLogs) (110456) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:13,476::__init__::232::IOProcess::(_processLogs) (110456) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:13,476::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:13,477::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:13,478::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:13,478::__init__::232::IOProcess::(_processLogs) (110457) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:13,478::__init__::232::IOProcess::(_processLogs) (110457) Queuing response >Dummy-148::DEBUG::2014-09-25 18:06:13,692::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.222908 s, 4.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:06:15,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:15,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175252 s, 58.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:15,855::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:15,856::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71710::DEBUG::2014-09-25 18:06:15,859::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:06:17,157::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:17,157::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:17,157::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:17,157::__init__::232::IOProcess::(_processLogs) (110458) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:17,172::__init__::232::IOProcess::(_processLogs) (110458) Queuing response >Thread-81::DEBUG::2014-09-25 18:06:17,173::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:06:17,192::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00890211 s, 41.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:17,193::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:17,193::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:17,194::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:17,194::__init__::232::IOProcess::(_processLogs) (110459) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:17,197::__init__::232::IOProcess::(_processLogs) (110459) Queuing response >Dummy-148::DEBUG::2014-09-25 18:06:17,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:17,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157704 s, 64.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:18,877::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:18,879::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71711::DEBUG::2014-09-25 18:06:18,882::task::595::Storage.TaskManager.Task::(_updateState) Task=`2ecd42c2-f821-4524-9e82-1e8650c8f9b0`::moving from state init -> state preparing >Thread-71711::INFO::2014-09-25 18:06:18,882::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71711::INFO::2014-09-25 18:06:18,883::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00903574', 'lastCheck': '5.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '20.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00890211', 'lastCheck': '1.7', 'valid': True}} >Thread-71711::DEBUG::2014-09-25 18:06:18,883::task::1191::Storage.TaskManager.Task::(prepare) Task=`2ecd42c2-f821-4524-9e82-1e8650c8f9b0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00903574', 'lastCheck': '5.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '20.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00890211', 'lastCheck': '1.7', 'valid': True}} >Thread-71711::DEBUG::2014-09-25 18:06:18,883::task::595::Storage.TaskManager.Task::(_updateState) Task=`2ecd42c2-f821-4524-9e82-1e8650c8f9b0`::moving from state preparing -> state finished >Thread-71711::DEBUG::2014-09-25 18:06:18,883::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71711::DEBUG::2014-09-25 18:06:18,884::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71711::DEBUG::2014-09-25 18:06:18,884::task::993::Storage.TaskManager.Task::(_decref) Task=`2ecd42c2-f821-4524-9e82-1e8650c8f9b0`::ref 0 aborting False >Thread-71711::DEBUG::2014-09-25 18:06:18,886::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:18,906::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:18,908::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71712::DEBUG::2014-09-25 18:06:18,909::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:19,724::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:19,725::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71713::DEBUG::2014-09-25 18:06:19,725::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71713::DEBUG::2014-09-25 18:06:19,727::task::595::Storage.TaskManager.Task::(_updateState) Task=`f3c317f9-dca9-44cb-b893-cb6934ed3a04`::moving from state init -> state preparing >Thread-71713::INFO::2014-09-25 18:06:19,727::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71713::INFO::2014-09-25 18:06:19,728::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71713::DEBUG::2014-09-25 18:06:19,729::task::1191::Storage.TaskManager.Task::(prepare) Task=`f3c317f9-dca9-44cb-b893-cb6934ed3a04`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71713::DEBUG::2014-09-25 18:06:19,729::task::595::Storage.TaskManager.Task::(_updateState) Task=`f3c317f9-dca9-44cb-b893-cb6934ed3a04`::moving from state preparing -> state finished >Thread-71713::DEBUG::2014-09-25 18:06:19,729::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71713::DEBUG::2014-09-25 18:06:19,730::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71713::DEBUG::2014-09-25 18:06:19,730::task::993::Storage.TaskManager.Task::(_decref) Task=`f3c317f9-dca9-44cb-b893-cb6934ed3a04`::ref 0 aborting False >Thread-71713::DEBUG::2014-09-25 18:06:19,730::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71713::DEBUG::2014-09-25 18:06:19,731::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:19,745::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71714::DEBUG::2014-09-25 18:06:19,746::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:06:19,746::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71714::DEBUG::2014-09-25 18:06:19,748::task::595::Storage.TaskManager.Task::(_updateState) Task=`538becee-003b-4ed1-b0d8-93e461af1d8d`::moving from state init -> state preparing >Thread-71714::INFO::2014-09-25 18:06:19,748::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71714::DEBUG::2014-09-25 18:06:19,749::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4eca8501-88bd-46e3-8357-dc4171d5f1d1`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71714::DEBUG::2014-09-25 18:06:19,749::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71714::DEBUG::2014-09-25 18:06:19,750::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71714::DEBUG::2014-09-25 18:06:19,750::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`4eca8501-88bd-46e3-8357-dc4171d5f1d1`::Granted request >Thread-71714::DEBUG::2014-09-25 18:06:19,751::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`538becee-003b-4ed1-b0d8-93e461af1d8d`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71714::DEBUG::2014-09-25 18:06:19,751::task::993::Storage.TaskManager.Task::(_decref) Task=`538becee-003b-4ed1-b0d8-93e461af1d8d`::ref 1 aborting False >Thread-71714::INFO::2014-09-25 18:06:19,755::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71714::DEBUG::2014-09-25 18:06:19,755::task::1191::Storage.TaskManager.Task::(prepare) Task=`538becee-003b-4ed1-b0d8-93e461af1d8d`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71714::DEBUG::2014-09-25 18:06:19,756::task::595::Storage.TaskManager.Task::(_updateState) Task=`538becee-003b-4ed1-b0d8-93e461af1d8d`::moving from state preparing -> state finished >Thread-71714::DEBUG::2014-09-25 18:06:19,756::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71714::DEBUG::2014-09-25 18:06:19,756::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71714::DEBUG::2014-09-25 18:06:19,756::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71714::DEBUG::2014-09-25 18:06:19,757::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71714::DEBUG::2014-09-25 18:06:19,757::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71714::DEBUG::2014-09-25 18:06:19,757::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71714::DEBUG::2014-09-25 18:06:19,758::task::993::Storage.TaskManager.Task::(_decref) Task=`538becee-003b-4ed1-b0d8-93e461af1d8d`::ref 0 aborting False >Thread-71714::DEBUG::2014-09-25 18:06:19,758::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71714::DEBUG::2014-09-25 18:06:19,758::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:19,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:19,836::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015584 s, 65.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:06:21,856::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:21,888::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205888 s, 49.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:21,980::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:21,982::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71715::DEBUG::2014-09-25 18:06:21,984::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:06:23,478::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:23,479::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:23,479::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:23,479::__init__::232::IOProcess::(_processLogs) (110460) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:23,482::__init__::232::IOProcess::(_processLogs) (110460) Queuing response >Thread-79::DEBUG::2014-09-25 18:06:23,483::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:06:23,495::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00210091 s, 268 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:23,497::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:23,497::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:23,497::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:23,497::__init__::232::IOProcess::(_processLogs) (110461) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:23,498::__init__::232::IOProcess::(_processLogs) (110461) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:23,499::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:23,499::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:23,500::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:23,500::__init__::232::IOProcess::(_processLogs) (110462) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:23,501::__init__::232::IOProcess::(_processLogs) (110462) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:23,501::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:23,502::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:23,502::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:23,502::__init__::232::IOProcess::(_processLogs) (110463) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:23,503::__init__::232::IOProcess::(_processLogs) (110463) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:23,503::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:23,504::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:23,505::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:23,505::__init__::232::IOProcess::(_processLogs) (110464) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:23,505::__init__::232::IOProcess::(_processLogs) (110464) Queuing response >Dummy-148::DEBUG::2014-09-25 18:06:23,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:23,939::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151776 s, 67.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:25,010::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:25,012::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71716::DEBUG::2014-09-25 18:06:25,014::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:25,959::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:25,996::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0247635 s, 41.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:27,199::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:27,200::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:27,200::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:27,200::__init__::232::IOProcess::(_processLogs) (110465) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:27,201::__init__::232::IOProcess::(_processLogs) (110465) Queuing response >Thread-81::DEBUG::2014-09-25 18:06:27,202::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:06:27,211::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00046584 s, 796 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:27,212::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:27,212::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:27,212::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:27,213::__init__::232::IOProcess::(_processLogs) (110466) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:27,213::__init__::232::IOProcess::(_processLogs) (110466) Queuing response >Dummy-148::DEBUG::2014-09-25 18:06:28,016::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:28,030::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:28,032::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71717::DEBUG::2014-09-25 18:06:28,035::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:28,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172606 s, 59.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:29,796::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71718::DEBUG::2014-09-25 18:06:29,797::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:06:29,798::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71718::DEBUG::2014-09-25 18:06:29,800::task::595::Storage.TaskManager.Task::(_updateState) Task=`de3ce01b-594c-496e-8588-c8f12ece4d2e`::moving from state init -> state preparing >Thread-71718::INFO::2014-09-25 18:06:29,800::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71718::INFO::2014-09-25 18:06:29,801::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71718::DEBUG::2014-09-25 18:06:29,801::task::1191::Storage.TaskManager.Task::(prepare) Task=`de3ce01b-594c-496e-8588-c8f12ece4d2e`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71718::DEBUG::2014-09-25 18:06:29,801::task::595::Storage.TaskManager.Task::(_updateState) Task=`de3ce01b-594c-496e-8588-c8f12ece4d2e`::moving from state preparing -> state finished >Thread-71718::DEBUG::2014-09-25 18:06:29,801::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71718::DEBUG::2014-09-25 18:06:29,802::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71718::DEBUG::2014-09-25 18:06:29,802::task::993::Storage.TaskManager.Task::(_decref) Task=`de3ce01b-594c-496e-8588-c8f12ece4d2e`::ref 0 aborting False >Thread-71718::DEBUG::2014-09-25 18:06:29,802::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71718::DEBUG::2014-09-25 18:06:29,803::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:29,818::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71719::DEBUG::2014-09-25 18:06:29,819::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:06:29,819::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71719::DEBUG::2014-09-25 18:06:29,821::task::595::Storage.TaskManager.Task::(_updateState) Task=`cecc2c99-6174-4994-8808-b898021fb3fd`::moving from state init -> state preparing >Thread-71719::INFO::2014-09-25 18:06:29,821::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71719::DEBUG::2014-09-25 18:06:29,822::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6f4c3512-acb0-400a-b16c-0deb1f0975db`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71719::DEBUG::2014-09-25 18:06:29,822::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71719::DEBUG::2014-09-25 18:06:29,823::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71719::DEBUG::2014-09-25 18:06:29,823::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`6f4c3512-acb0-400a-b16c-0deb1f0975db`::Granted request >Thread-71719::DEBUG::2014-09-25 18:06:29,823::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`cecc2c99-6174-4994-8808-b898021fb3fd`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71719::DEBUG::2014-09-25 18:06:29,824::task::993::Storage.TaskManager.Task::(_decref) Task=`cecc2c99-6174-4994-8808-b898021fb3fd`::ref 1 aborting False >Thread-71719::INFO::2014-09-25 18:06:29,828::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71719::DEBUG::2014-09-25 18:06:29,828::task::1191::Storage.TaskManager.Task::(prepare) Task=`cecc2c99-6174-4994-8808-b898021fb3fd`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71719::DEBUG::2014-09-25 18:06:29,829::task::595::Storage.TaskManager.Task::(_updateState) Task=`cecc2c99-6174-4994-8808-b898021fb3fd`::moving from state preparing -> state finished >Thread-71719::DEBUG::2014-09-25 18:06:29,829::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71719::DEBUG::2014-09-25 18:06:29,829::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71719::DEBUG::2014-09-25 18:06:29,829::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71719::DEBUG::2014-09-25 18:06:29,830::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71719::DEBUG::2014-09-25 18:06:29,830::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71719::DEBUG::2014-09-25 18:06:29,830::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71719::DEBUG::2014-09-25 18:06:29,830::task::993::Storage.TaskManager.Task::(_decref) Task=`cecc2c99-6174-4994-8808-b898021fb3fd`::ref 0 aborting False >Thread-71719::DEBUG::2014-09-25 18:06:29,831::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71719::DEBUG::2014-09-25 18:06:29,831::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:30,062::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:30,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0227723 s, 45.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:31,063::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:31,065::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71720::DEBUG::2014-09-25 18:06:31,067::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:32,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:32,221::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.012959 s, 79.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:33,505::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:33,506::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:33,506::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:33,506::__init__::232::IOProcess::(_processLogs) (110467) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:33,507::__init__::232::IOProcess::(_processLogs) (110467) Queuing response >Thread-79::DEBUG::2014-09-25 18:06:33,508::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:06:33,517::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000580266 s, 969 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:33,518::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:33,518::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:33,519::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:33,519::__init__::232::IOProcess::(_processLogs) (110468) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:33,520::__init__::232::IOProcess::(_processLogs) (110468) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:33,521::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:33,522::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:33,522::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:33,523::__init__::232::IOProcess::(_processLogs) (110469) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:33,523::__init__::232::IOProcess::(_processLogs) (110469) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:33,523::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:33,524::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:33,524::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:33,524::__init__::232::IOProcess::(_processLogs) (110470) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:33,525::__init__::232::IOProcess::(_processLogs) (110470) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:33,525::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:33,526::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:33,526::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:33,526::__init__::232::IOProcess::(_processLogs) (110471) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:33,526::__init__::232::IOProcess::(_processLogs) (110471) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:34,084::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:34,085::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71721::DEBUG::2014-09-25 18:06:34,088::task::595::Storage.TaskManager.Task::(_updateState) Task=`a0bd6c2e-5ffe-4dc3-89ce-1c8f2c07771c`::moving from state init -> state preparing >Thread-71721::INFO::2014-09-25 18:06:34,089::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71721::INFO::2014-09-25 18:06:34,089::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000580266', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '36.1', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00046584', 'lastCheck': '6.9', 'valid': True}} >Thread-71721::DEBUG::2014-09-25 18:06:34,089::task::1191::Storage.TaskManager.Task::(prepare) Task=`a0bd6c2e-5ffe-4dc3-89ce-1c8f2c07771c`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000580266', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '36.1', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00046584', 'lastCheck': '6.9', 'valid': True}} >Thread-71721::DEBUG::2014-09-25 18:06:34,090::task::595::Storage.TaskManager.Task::(_updateState) Task=`a0bd6c2e-5ffe-4dc3-89ce-1c8f2c07771c`::moving from state preparing -> state finished >Thread-71721::DEBUG::2014-09-25 18:06:34,090::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71721::DEBUG::2014-09-25 18:06:34,090::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71721::DEBUG::2014-09-25 18:06:34,090::task::993::Storage.TaskManager.Task::(_decref) Task=`a0bd6c2e-5ffe-4dc3-89ce-1c8f2c07771c`::ref 0 aborting False >Thread-71721::DEBUG::2014-09-25 18:06:34,093::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:34,130::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:34,131::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71722::DEBUG::2014-09-25 18:06:34,133::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:34,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:34,275::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202722 s, 50.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:06:36,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:36,332::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0216963 s, 47.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:37,199::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:37,201::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71723::DEBUG::2014-09-25 18:06:37,203::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:06:37,215::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:37,216::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:37,216::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:37,216::__init__::232::IOProcess::(_processLogs) (110472) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:37,217::__init__::232::IOProcess::(_processLogs) (110472) Queuing response >Thread-81::DEBUG::2014-09-25 18:06:37,217::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:06:37,235::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00798099 s, 46.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:37,237::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:37,237::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:37,237::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:37,237::__init__::232::IOProcess::(_processLogs) (110473) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:37,242::__init__::232::IOProcess::(_processLogs) (110473) Queuing response >Dummy-148::DEBUG::2014-09-25 18:06:38,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:38,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0206799 s, 49.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:39,860::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71724::DEBUG::2014-09-25 18:06:39,862::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:06:39,862::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71724::DEBUG::2014-09-25 18:06:39,864::task::595::Storage.TaskManager.Task::(_updateState) Task=`318fe27f-0cc0-438b-833a-8de4007e0809`::moving from state init -> state preparing >Thread-71724::INFO::2014-09-25 18:06:39,865::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71724::INFO::2014-09-25 18:06:39,865::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71724::DEBUG::2014-09-25 18:06:39,865::task::1191::Storage.TaskManager.Task::(prepare) Task=`318fe27f-0cc0-438b-833a-8de4007e0809`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71724::DEBUG::2014-09-25 18:06:39,866::task::595::Storage.TaskManager.Task::(_updateState) Task=`318fe27f-0cc0-438b-833a-8de4007e0809`::moving from state preparing -> state finished >Thread-71724::DEBUG::2014-09-25 18:06:39,866::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71724::DEBUG::2014-09-25 18:06:39,866::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71724::DEBUG::2014-09-25 18:06:39,866::task::993::Storage.TaskManager.Task::(_decref) Task=`318fe27f-0cc0-438b-833a-8de4007e0809`::ref 0 aborting False >Thread-71724::DEBUG::2014-09-25 18:06:39,867::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71724::DEBUG::2014-09-25 18:06:39,867::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:39,881::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71725::DEBUG::2014-09-25 18:06:39,882::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:06:39,882::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71725::DEBUG::2014-09-25 18:06:39,884::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f75ac96-d9a0-4d58-869f-5cb605a05c2b`::moving from state init -> state preparing >Thread-71725::INFO::2014-09-25 18:06:39,885::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71725::DEBUG::2014-09-25 18:06:39,885::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1211821c-7754-43d6-b916-76ec8435cd9a`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71725::DEBUG::2014-09-25 18:06:39,886::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71725::DEBUG::2014-09-25 18:06:39,886::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71725::DEBUG::2014-09-25 18:06:39,886::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1211821c-7754-43d6-b916-76ec8435cd9a`::Granted request >Thread-71725::DEBUG::2014-09-25 18:06:39,887::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`0f75ac96-d9a0-4d58-869f-5cb605a05c2b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71725::DEBUG::2014-09-25 18:06:39,887::task::993::Storage.TaskManager.Task::(_decref) Task=`0f75ac96-d9a0-4d58-869f-5cb605a05c2b`::ref 1 aborting False >Thread-71725::INFO::2014-09-25 18:06:39,892::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71725::DEBUG::2014-09-25 18:06:39,892::task::1191::Storage.TaskManager.Task::(prepare) Task=`0f75ac96-d9a0-4d58-869f-5cb605a05c2b`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71725::DEBUG::2014-09-25 18:06:39,892::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f75ac96-d9a0-4d58-869f-5cb605a05c2b`::moving from state preparing -> state finished >Thread-71725::DEBUG::2014-09-25 18:06:39,892::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71725::DEBUG::2014-09-25 18:06:39,893::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71725::DEBUG::2014-09-25 18:06:39,893::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71725::DEBUG::2014-09-25 18:06:39,893::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71725::DEBUG::2014-09-25 18:06:39,893::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71725::DEBUG::2014-09-25 18:06:39,894::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71725::DEBUG::2014-09-25 18:06:39,894::task::993::Storage.TaskManager.Task::(_decref) Task=`0f75ac96-d9a0-4d58-869f-5cb605a05c2b`::ref 0 aborting False >Thread-71725::DEBUG::2014-09-25 18:06:39,894::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71725::DEBUG::2014-09-25 18:06:39,895::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:40,223::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:40,226::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71726::DEBUG::2014-09-25 18:06:40,231::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:40,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:40,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168846 s, 60.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:06:42,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:42,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182702 s, 56.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:43,250::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:43,252::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71727::DEBUG::2014-09-25 18:06:43,254::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:06:43,528::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:43,529::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:43,529::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:43,529::__init__::232::IOProcess::(_processLogs) (110474) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:43,529::__init__::232::IOProcess::(_processLogs) (110474) Queuing response >Thread-79::DEBUG::2014-09-25 18:06:43,531::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:06:43,540::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000410001 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:43,541::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:43,541::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:43,541::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:43,542::__init__::232::IOProcess::(_processLogs) (110475) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:43,542::__init__::232::IOProcess::(_processLogs) (110475) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:43,544::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:43,544::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:43,545::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:43,545::__init__::232::IOProcess::(_processLogs) (110476) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:43,546::__init__::232::IOProcess::(_processLogs) (110476) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:43,546::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:43,547::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:43,547::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:43,548::__init__::232::IOProcess::(_processLogs) (110477) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:43,548::__init__::232::IOProcess::(_processLogs) (110477) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:43,548::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:43,549::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:43,550::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:43,550::__init__::232::IOProcess::(_processLogs) (110478) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:43,550::__init__::232::IOProcess::(_processLogs) (110478) Queuing response >Dummy-148::DEBUG::2014-09-25 18:06:44,505::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:44,531::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170386 s, 60.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:46,280::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:46,283::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71728::DEBUG::2014-09-25 18:06:46,288::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:46,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:46,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215678 s, 47.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:47,244::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:47,244::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:47,245::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:47,245::__init__::232::IOProcess::(_processLogs) (110479) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:47,245::__init__::232::IOProcess::(_processLogs) (110479) Queuing response >Thread-81::DEBUG::2014-09-25 18:06:47,247::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:06:47,267::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0055538 s, 66.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:47,268::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:47,269::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:47,269::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:47,269::__init__::232::IOProcess::(_processLogs) (110480) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:47,270::__init__::232::IOProcess::(_processLogs) (110480) Queuing response >Dummy-148::DEBUG::2014-09-25 18:06:48,694::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:48,716::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135493 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:49,309::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:49,311::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71729::DEBUG::2014-09-25 18:06:49,313::task::595::Storage.TaskManager.Task::(_updateState) Task=`4d3f69a8-dc01-451d-959e-5a98fe75d313`::moving from state init -> state preparing >Thread-71729::INFO::2014-09-25 18:06:49,314::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71729::INFO::2014-09-25 18:06:49,314::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000410001', 'lastCheck': '5.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '51.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0055538', 'lastCheck': '2.0', 'valid': True}} >Thread-71729::DEBUG::2014-09-25 18:06:49,314::task::1191::Storage.TaskManager.Task::(prepare) Task=`4d3f69a8-dc01-451d-959e-5a98fe75d313`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000410001', 'lastCheck': '5.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '51.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0055538', 'lastCheck': '2.0', 'valid': True}} >Thread-71729::DEBUG::2014-09-25 18:06:49,315::task::595::Storage.TaskManager.Task::(_updateState) Task=`4d3f69a8-dc01-451d-959e-5a98fe75d313`::moving from state preparing -> state finished >Thread-71729::DEBUG::2014-09-25 18:06:49,315::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71729::DEBUG::2014-09-25 18:06:49,315::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71729::DEBUG::2014-09-25 18:06:49,315::task::993::Storage.TaskManager.Task::(_decref) Task=`4d3f69a8-dc01-451d-959e-5a98fe75d313`::ref 0 aborting False >Thread-71729::DEBUG::2014-09-25 18:06:49,318::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:49,363::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:49,364::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71730::DEBUG::2014-09-25 18:06:49,366::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:49,924::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71731::DEBUG::2014-09-25 18:06:49,925::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:06:49,925::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71731::DEBUG::2014-09-25 18:06:49,927::task::595::Storage.TaskManager.Task::(_updateState) Task=`e55ee73b-19fd-4754-83f0-7e38b353543a`::moving from state init -> state preparing >Thread-71731::INFO::2014-09-25 18:06:49,928::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71731::INFO::2014-09-25 18:06:49,928::logUtils::47::dispatcher::(wrapper) Run and protect: getSpmStatus, Return response: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71731::DEBUG::2014-09-25 18:06:49,928::task::1191::Storage.TaskManager.Task::(prepare) Task=`e55ee73b-19fd-4754-83f0-7e38b353543a`::finished: {'spm_st': {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7}} >Thread-71731::DEBUG::2014-09-25 18:06:49,929::task::595::Storage.TaskManager.Task::(_updateState) Task=`e55ee73b-19fd-4754-83f0-7e38b353543a`::moving from state preparing -> state finished >Thread-71731::DEBUG::2014-09-25 18:06:49,929::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71731::DEBUG::2014-09-25 18:06:49,929::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71731::DEBUG::2014-09-25 18:06:49,929::task::993::Storage.TaskManager.Task::(_decref) Task=`e55ee73b-19fd-4754-83f0-7e38b353543a`::ref 0 aborting False >Thread-71731::DEBUG::2014-09-25 18:06:49,930::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getSpmStatus' in bridge with {'spmId': 2, 'spmStatus': 'SPM', 'spmLver': 7} >Thread-71731::DEBUG::2014-09-25 18:06:49,930::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:49,944::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71732::DEBUG::2014-09-25 18:06:49,945::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getInfo' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:06:49,945::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71732::DEBUG::2014-09-25 18:06:49,947::task::595::Storage.TaskManager.Task::(_updateState) Task=`a33eed26-4b4d-423c-b01e-fef5e843e29b`::moving from state init -> state preparing >Thread-71732::INFO::2014-09-25 18:06:49,947::logUtils::44::dispatcher::(wrapper) Run and protect: getStoragePoolInfo(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71732::DEBUG::2014-09-25 18:06:49,948::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`315e2071-2c4c-45da-abb8-17f088b146f2`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '2616' at 'getStoragePoolInfo' >Thread-71732::DEBUG::2014-09-25 18:06:49,948::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71732::DEBUG::2014-09-25 18:06:49,949::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71732::DEBUG::2014-09-25 18:06:49,949::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`315e2071-2c4c-45da-abb8-17f088b146f2`::Granted request >Thread-71732::DEBUG::2014-09-25 18:06:49,949::task::827::Storage.TaskManager.Task::(resourceAcquired) Task=`a33eed26-4b4d-423c-b01e-fef5e843e29b`::_resourcesAcquired: Storage.00000002-0002-0002-0002-0000000001ad (shared) >Thread-71732::DEBUG::2014-09-25 18:06:49,950::task::993::Storage.TaskManager.Task::(_decref) Task=`a33eed26-4b4d-423c-b01e-fef5e843e29b`::ref 1 aborting False >Thread-71732::INFO::2014-09-25 18:06:49,954::logUtils::47::dispatcher::(wrapper) Run and protect: getStoragePoolInfo, Return response: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71732::DEBUG::2014-09-25 18:06:49,954::task::1191::Storage.TaskManager.Task::(prepare) Task=`a33eed26-4b4d-423c-b01e-fef5e843e29b`::finished: {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71732::DEBUG::2014-09-25 18:06:49,954::task::595::Storage.TaskManager.Task::(_updateState) Task=`a33eed26-4b4d-423c-b01e-fef5e843e29b`::moving from state preparing -> state finished >Thread-71732::DEBUG::2014-09-25 18:06:49,955::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {u'Storage.00000002-0002-0002-0002-0000000001ad': < ResourceRef 'Storage.00000002-0002-0002-0002-0000000001ad', isValid: 'True' obj: 'None'>} >Thread-71732::DEBUG::2014-09-25 18:06:49,955::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71732::DEBUG::2014-09-25 18:06:49,955::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71732::DEBUG::2014-09-25 18:06:49,955::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71732::DEBUG::2014-09-25 18:06:49,956::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71732::DEBUG::2014-09-25 18:06:49,956::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71732::DEBUG::2014-09-25 18:06:49,956::task::993::Storage.TaskManager.Task::(_decref) Task=`a33eed26-4b4d-423c-b01e-fef5e843e29b`::ref 0 aborting False >Thread-71732::DEBUG::2014-09-25 18:06:49,957::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.getInfo' in bridge with {'info': {'name': 'Default', 'isoprefix': '', 'pool_status': 'connected', 'lver': 7, 'domains': 'eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'master_uuid': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'version': '0', 'spm_id': 2, 'type': 'NFS', 'master_ver': 1}, 'dominfo': {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'status': 'Active', 'isoprefix': '', 'alerts': [], 'version': -1}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'status': 'Active', 'diskfree': '75196530688', 'isoprefix': '', 'alerts': [], 'disktotal': '422758580224', 'version': 0}}} >Thread-71732::DEBUG::2014-09-25 18:06:49,957::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:50,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:50,974::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.22169 s, 4.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:52,454::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:52,456::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71733::DEBUG::2014-09-25 18:06:52,460::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:52,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:53,023::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181659 s, 56.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:53,551::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:53,551::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:53,551::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:53,551::__init__::232::IOProcess::(_processLogs) (110481) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:53,553::__init__::232::IOProcess::(_processLogs) (110481) Queuing response >Thread-79::DEBUG::2014-09-25 18:06:53,553::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:06:53,570::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00256711 s, 219 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:53,572::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:53,572::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:53,572::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:53,572::__init__::232::IOProcess::(_processLogs) (110482) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:53,577::__init__::232::IOProcess::(_processLogs) (110482) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:53,578::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:53,578::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:53,579::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:53,579::__init__::232::IOProcess::(_processLogs) (110483) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:53,583::__init__::232::IOProcess::(_processLogs) (110483) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:53,584::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:53,584::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:53,584::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:53,585::__init__::232::IOProcess::(_processLogs) (110484) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:53,591::__init__::232::IOProcess::(_processLogs) (110484) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:53,592::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:53,592::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:53,592::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:53,593::__init__::232::IOProcess::(_processLogs) (110485) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:53,598::__init__::232::IOProcess::(_processLogs) (110485) Queuing response >Dummy-148::DEBUG::2014-09-25 18:06:55,056::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:55,088::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017972 s, 57.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:55,479::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:55,481::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71734::DEBUG::2014-09-25 18:06:55,483::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:06:57,108::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:57,133::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141968 s, 72.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:57,272::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:57,272::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:57,273::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:57,273::__init__::232::IOProcess::(_processLogs) (110486) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:57,278::__init__::232::IOProcess::(_processLogs) (110486) Queuing response >Thread-81::DEBUG::2014-09-25 18:06:57,279::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:06:57,298::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00656396 s, 56.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:06:57,299::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:57,300::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:57,300::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:57,300::__init__::232::IOProcess::(_processLogs) (110487) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:06:57,304::__init__::232::IOProcess::(_processLogs) (110487) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:58,506::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:58,508::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71735::DEBUG::2014-09-25 18:06:58,510::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-71693::DEBUG::2014-09-25 18:06:58,756::hsm::2441::Storage.HSM::(connectStorageServer) prefetch failed: {u'3573f707-d0c4-4dc9-b09f-dca9a3506a4f': <function findDomain at 0x14be758>, u'fc587b6a-bac3-4668-99de-7f6fd6acce7f': <function findDomain at 0x14be758>, u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': <function findDomain at 0x14be758>} >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2438, in connectStorageServer > doms = self.__prefetchDomains(domType, conObj) > File "/usr/share/vdsm/storage/hsm.py", line 2375, in __prefetchDomains > goop.glob.glob(os.path.join(lPath, uuidPatern))) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-71693::DEBUG::2014-09-25 18:06:58,758::hsm::2452::Storage.HSM::(connectStorageServer) knownSDs: {3573f707-d0c4-4dc9-b09f-dca9a3506a4f: storage.nfsSD.findDomain, fc587b6a-bac3-4668-99de-7f6fd6acce7f: storage.nfsSD.findDomain, eb633636-3019-47a9-9f5a-fc81f8dfd7dc: storage.nfsSD.findDomain} >Thread-71693::DEBUG::2014-09-25 18:06:58,762::hsm::2372::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35 >Thread-29::DEBUG::2014-09-25 18:06:58,762::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:58,763::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:58,763::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:58,763::__init__::232::IOProcess::(_processLogs) (110488) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:58,766::__init__::232::IOProcess::(_processLogs) (110488) Queuing response >Thread-71693::DEBUG::2014-09-25 18:06:58,767::hsm::2396::Storage.HSM::(__prefetchDomains) Found SD uuids: (u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc',) >Thread-71693::DEBUG::2014-09-25 18:06:58,768::hsm::2452::Storage.HSM::(connectStorageServer) knownSDs: {3573f707-d0c4-4dc9-b09f-dca9a3506a4f: storage.nfsSD.findDomain, fc587b6a-bac3-4668-99de-7f6fd6acce7f: storage.nfsSD.findDomain, eb633636-3019-47a9-9f5a-fc81f8dfd7dc: storage.nfsSD.findDomain} >Thread-71693::INFO::2014-09-25 18:06:58,768::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': u'6d77ae90-0a92-4f76-9980-cb7eceb7df64'}, {'status': 0, 'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8'}, {'status': 0, 'id': u'd7af8c51-78e6-44db-88d2-f765dc508251'}]} >Thread-71693::DEBUG::2014-09-25 18:06:58,768::task::1191::Storage.TaskManager.Task::(prepare) Task=`38036347-d013-4b56-8e1b-92c4e200428f`::finished: {'statuslist': [{'status': 0, 'id': u'6d77ae90-0a92-4f76-9980-cb7eceb7df64'}, {'status': 0, 'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8'}, {'status': 0, 'id': u'd7af8c51-78e6-44db-88d2-f765dc508251'}]} >Thread-71693::DEBUG::2014-09-25 18:06:58,769::task::595::Storage.TaskManager.Task::(_updateState) Task=`38036347-d013-4b56-8e1b-92c4e200428f`::moving from state preparing -> state finished >Thread-71693::DEBUG::2014-09-25 18:06:58,769::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71693::DEBUG::2014-09-25 18:06:58,769::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71693::DEBUG::2014-09-25 18:06:58,769::task::993::Storage.TaskManager.Task::(_decref) Task=`38036347-d013-4b56-8e1b-92c4e200428f`::ref 0 aborting False >Thread-71693::DEBUG::2014-09-25 18:06:58,770::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.connectStorageServer' in bridge with [{'status': 0, 'id': u'6d77ae90-0a92-4f76-9980-cb7eceb7df64'}, {'status': 0, 'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8'}, {'status': 0, 'id': u'd7af8c51-78e6-44db-88d2-f765dc508251'}] >Thread-71693::DEBUG::2014-09-25 18:06:58,770::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:58,786::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:06:58,787::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71736::DEBUG::2014-09-25 18:06:58,787::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.connect' in bridge with {u'masterVersion': 1, u'domainDict': {u'3573f707-d0c4-4dc9-b09f-dca9a3506a4f': u'active', u'fc587b6a-bac3-4668-99de-7f6fd6acce7f': u'active', u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': u'active'}, u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad', u'scsiKey': u'00000002-0002-0002-0002-0000000001ad', u'masterSdUUID': u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', u'hostID': 2} >Thread-71736::DEBUG::2014-09-25 18:06:58,789::task::595::Storage.TaskManager.Task::(_updateState) Task=`e73af8b8-c6af-4271-83d8-95e8fcc6bd90`::moving from state init -> state preparing >Thread-71736::INFO::2014-09-25 18:06:58,789::logUtils::44::dispatcher::(wrapper) Run and protect: connectStoragePool(spUUID=u'00000002-0002-0002-0002-0000000001ad', hostID=2, msdUUID=u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc', masterVersion=1, domainsMap={u'3573f707-d0c4-4dc9-b09f-dca9a3506a4f': u'active', u'fc587b6a-bac3-4668-99de-7f6fd6acce7f': u'active', u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': u'active'}, options=None) >Thread-71736::DEBUG::2014-09-25 18:06:58,790::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f0777be0-6b1d-4f51-9188-5658e1590da9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1011' at 'connectStoragePool' >Thread-71736::DEBUG::2014-09-25 18:06:58,790::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.HsmDomainMonitorLock' for lock type 'exclusive' >Thread-71736::DEBUG::2014-09-25 18:06:58,791::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.HsmDomainMonitorLock' is free. Now locking as 'exclusive' (1 active user) >Thread-71736::DEBUG::2014-09-25 18:06:58,791::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.HsmDomainMonitorLock`ReqID=`f0777be0-6b1d-4f51-9188-5658e1590da9`::Granted request >Thread-71736::DEBUG::2014-09-25 18:06:58,792::resourceManager::198::Storage.ResourceManager.Request::(__init__) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1fd325ee-c4c6-4adf-b38a-05ea8eec48d9`::Request was made in '/usr/share/vdsm/storage/hsm.py' line '1049' at '_connectStoragePool' >Thread-71736::DEBUG::2014-09-25 18:06:58,792::resourceManager::542::Storage.ResourceManager::(registerResource) Trying to register resource 'Storage.00000002-0002-0002-0002-0000000001ad' for lock type 'shared' >Thread-71736::DEBUG::2014-09-25 18:06:58,792::resourceManager::601::Storage.ResourceManager::(registerResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free. Now locking as 'shared' (1 active user) >Thread-71736::DEBUG::2014-09-25 18:06:58,793::resourceManager::238::Storage.ResourceManager.Request::(grant) ResName=`Storage.00000002-0002-0002-0002-0000000001ad`ReqID=`1fd325ee-c4c6-4adf-b38a-05ea8eec48d9`::Granted request >Thread-71736::DEBUG::2014-09-25 18:06:58,793::spbackends::485::Storage.StoragePoolMemoryBackend::(updateVersionAndDomains) updating domain version to 1 and domains map to {u'3573f707-d0c4-4dc9-b09f-dca9a3506a4f': u'active', u'fc587b6a-bac3-4668-99de-7f6fd6acce7f': u'active', u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': u'active'} >Thread-71736::INFO::2014-09-25 18:06:58,793::spbackends::433::Storage.StoragePoolMemoryBackend::(setDomainsMap) new storage pool master version 1 and domains map {u'3573f707-d0c4-4dc9-b09f-dca9a3506a4f': u'Active', u'fc587b6a-bac3-4668-99de-7f6fd6acce7f': u'Active', u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': u'Active'} >Thread-71736::INFO::2014-09-25 18:06:58,794::sp::133::Storage.StoragePool::(setBackend) updating pool 00000002-0002-0002-0002-0000000001ad backend from type StoragePoolDiskBackend instance 0x7fee7c0af3b0 to type StoragePoolMemoryBackend instance 0x7fee88153f18 >Thread-71736::DEBUG::2014-09-25 18:06:58,794::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,794::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,795::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,795::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,795::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,795::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,796::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-71736::DEBUG::2014-09-25 18:06:58,796::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-71736::DEBUG::2014-09-25 18:06:58,796::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-71736::DEBUG::2014-09-25 18:06:58,796::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-71736::DEBUG::2014-09-25 18:06:58,797::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-71736::DEBUG::2014-09-25 18:06:58,797::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) >Thread-71736::DEBUG::2014-09-25 18:06:58,821::misc::751::Storage.SamplingMethod::(__call__) Returning last result >Thread-71736::DEBUG::2014-09-25 18:06:58,824::multipath::110::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /sbin/multipath (cwd None) >Thread-71736::DEBUG::2014-09-25 18:06:58,907::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-71736::DEBUG::2014-09-25 18:06:58,908::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,909::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,909::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,909::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,910::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,910::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,910::misc::751::Storage.SamplingMethod::(__call__) Returning last result >Thread-29::DEBUG::2014-09-25 18:06:58,914::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:58,915::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:58,915::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:58,915::__init__::232::IOProcess::(_processLogs) (110489) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:58,916::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:58,916::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:58,916::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:58,916::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:58,917::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:58,917::__init__::232::IOProcess::(_processLogs) (110490) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:58,917::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:58,917::__init__::232::IOProcess::(_processLogs) (110491) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:58,920::__init__::232::IOProcess::(_processLogs) (110489) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:58,923::__init__::232::IOProcess::(_processLogs) (110491) Queuing response >Thread-71736::DEBUG::2014-09-25 18:06:58,927::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 18:06:58,928::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:58,928::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:58,928::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:58,928::__init__::232::IOProcess::(_processLogs) (110492) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 18:06:58,960::__init__::232::IOProcess::(_processLogs) (110492) Queuing response >Thread-71736::DEBUG::2014-09-25 18:06:58,960::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:06:58,961::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:58,961::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:58,961::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:58,962::__init__::232::IOProcess::(_processLogs) (110493) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 18:06:58,972::__init__::232::IOProcess::(_processLogs) (110493) Queuing response >Thread-71736::DEBUG::2014-09-25 18:06:58,972::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 18:06:58,973::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:58,974::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:58,974::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:58,975::__init__::232::IOProcess::(_processLogs) (110494) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:06:58,975::__init__::232::IOProcess::(_processLogs) (110494) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:58,975::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:58,976::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:58,976::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:58,976::__init__::232::IOProcess::(_processLogs) (110495) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:58,986::__init__::232::IOProcess::(_processLogs) (110495) Queuing response >Thread-71736::DEBUG::2014-09-25 18:06:58,986::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-71736::INFO::2014-09-25 18:06:58,986::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-71736::INFO::2014-09-25 18:06:58,987::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-71736::DEBUG::2014-09-25 18:06:58,987::sp::1390::Storage.StoragePool::(setMasterDomain) Master domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc verified, version 1 >Thread-71736::DEBUG::2014-09-25 18:06:58,987::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sp.updateMonitoringThreads) >Thread-71736::DEBUG::2014-09-25 18:06:58,988::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-71736::DEBUG::2014-09-25 18:06:58,988::misc::751::Storage.SamplingMethod::(__call__) Returning last result >Thread-71736::DEBUG::2014-09-25 18:06:58,988::fileUtils::142::Storage.fileUtils::(createdir) Creating directory: /rhev/data-center/00000002-0002-0002-0002-0000000001ad >Thread-71736::WARNING::2014-09-25 18:06:58,988::fileUtils::149::Storage.fileUtils::(createdir) Dir /rhev/data-center/00000002-0002-0002-0002-0000000001ad already exists >Thread-71736::DEBUG::2014-09-25 18:06:58,989::lvm::365::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' got the operation mutex >Thread-71736::DEBUG::2014-09-25 18:06:58,990::lvm::288::Storage.Misc.excCmd::(cmd) /usr/bin/sudo -n /sbin/lvm vgs --config ' devices { preferred_names = ["^/dev/mapper/"] ignore_suspended_devices=1 write_cache_state=0 disable_after_error_count=3 obtain_device_list_from_udev=0 filter = [ '\''r|.*|'\'' ] } global { locking_type=1 prioritise_write_locks=1 wait_for_locks=1 use_lvmetad=0 } backup { retain_min = 50 retain_days = 0 } ' --noheadings --units b --nosuffix --separator '|' --ignoreskippedcluster -o uuid,name,attr,size,free,extent_size,extent_count,free_count,tags,vg_mda_size,vg_mda_free,lv_count,pv_count,pv_name 3573f707-d0c4-4dc9-b09f-dca9a3506a4f fc587b6a-bac3-4668-99de-7f6fd6acce7f (cwd None) >Thread-71736::DEBUG::2014-09-25 18:06:59,038::lvm::288::Storage.Misc.excCmd::(cmd) FAILED: <err> = ' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!\n Volume group "3573f707-d0c4-4dc9-b09f-dca9a3506a4f" not found\n Skipping volume group 3573f707-d0c4-4dc9-b09f-dca9a3506a4f\n Volume group "fc587b6a-bac3-4668-99de-7f6fd6acce7f" not found\n Skipping volume group fc587b6a-bac3-4668-99de-7f6fd6acce7f\n'; <rc> = 5 >Thread-71736::WARNING::2014-09-25 18:06:59,039::lvm::370::Storage.LVM::(_reloadvgs) lvm vgs failed: 5 [] [' WARNING: lvmetad is running but disabled. Restart lvmetad before enabling it!', ' Volume group "3573f707-d0c4-4dc9-b09f-dca9a3506a4f" not found', ' Skipping volume group 3573f707-d0c4-4dc9-b09f-dca9a3506a4f', ' Volume group "fc587b6a-bac3-4668-99de-7f6fd6acce7f" not found', ' Skipping volume group fc587b6a-bac3-4668-99de-7f6fd6acce7f'] >Thread-71736::DEBUG::2014-09-25 18:06:59,040::lvm::407::Storage.OperationMutex::(_reloadvgs) Operation 'lvm reload operation' released the operation mutex >Thread-29::DEBUG::2014-09-25 18:06:59,044::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:59,044::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:59,044::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:59,045::__init__::232::IOProcess::(_processLogs) (110496) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:59,045::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:59,045::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:59,046::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:06:59,046::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:59,046::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:06:59,046::__init__::232::IOProcess::(_processLogs) (110497) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:59,047::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:06:59,047::__init__::232::IOProcess::(_processLogs) (110498) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:06:59,049::__init__::232::IOProcess::(_processLogs) (110496) Queuing response >Thread-29::DEBUG::2014-09-25 18:06:59,051::__init__::232::IOProcess::(_processLogs) (110498) Queuing response >Dummy-148::DEBUG::2014-09-25 18:06:59,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:06:59,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209376 s, 48.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:06:59,986::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71743::DEBUG::2014-09-25 18:06:59,987::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:06:59,987::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71743::DEBUG::2014-09-25 18:06:59,990::task::595::Storage.TaskManager.Task::(_updateState) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::moving from state init -> state preparing >Thread-71743::INFO::2014-09-25 18:06:59,990::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71743::ERROR::2014-09-25 18:06:59,990::task::866::Storage.TaskManager.Task::(_setError) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71743::DEBUG::2014-09-25 18:06:59,993::task::885::Storage.TaskManager.Task::(_run) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::Task._run: 931ab89c-1abc-4998-9c90-9a6f28cad1c2 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71743::DEBUG::2014-09-25 18:06:59,993::task::1217::Storage.TaskManager.Task::(stop) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::stopping in state preparing (force False) >Thread-71743::DEBUG::2014-09-25 18:06:59,994::task::993::Storage.TaskManager.Task::(_decref) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::ref 1 aborting True >Thread-71743::INFO::2014-09-25 18:06:59,994::task::1171::Storage.TaskManager.Task::(prepare) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::aborting: Task is aborted: u'' - code 100 >Thread-71743::DEBUG::2014-09-25 18:06:59,994::task::1176::Storage.TaskManager.Task::(prepare) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::Prepare: aborted: >Thread-71743::DEBUG::2014-09-25 18:06:59,994::task::993::Storage.TaskManager.Task::(_decref) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::ref 0 aborting True >Thread-71743::DEBUG::2014-09-25 18:06:59,994::task::928::Storage.TaskManager.Task::(_doAbort) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::Task._doAbort: force False >Thread-71743::DEBUG::2014-09-25 18:06:59,995::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71743::DEBUG::2014-09-25 18:06:59,995::task::595::Storage.TaskManager.Task::(_updateState) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::moving from state preparing -> state aborting >Thread-71743::DEBUG::2014-09-25 18:06:59,995::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::_aborting: recover policy none >Thread-71743::DEBUG::2014-09-25 18:06:59,995::task::595::Storage.TaskManager.Task::(_updateState) Task=`931ab89c-1abc-4998-9c90-9a6f28cad1c2`::moving from state aborting -> state failed >Thread-71743::DEBUG::2014-09-25 18:06:59,996::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71743::DEBUG::2014-09-25 18:06:59,996::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71743::ERROR::2014-09-25 18:06:59,996::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71743::DEBUG::2014-09-25 18:06:59,997::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:07:00,074::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71744::DEBUG::2014-09-25 18:07:00,076::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getAllTasksStatuses' in bridge with {} >JsonRpcServer::DEBUG::2014-09-25 18:07:00,076::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71744::DEBUG::2014-09-25 18:07:00,078::task::595::Storage.TaskManager.Task::(_updateState) Task=`32dcecd4-8b73-4d05-8d5c-5ed5db76f4d1`::moving from state init -> state preparing >Thread-71744::INFO::2014-09-25 18:07:00,078::logUtils::44::dispatcher::(wrapper) Run and protect: getAllTasksStatuses(spUUID=None, options=None) >Thread-71744::DEBUG::2014-09-25 18:07:00,078::taskManager::112::Storage.TaskManager::(getAllTasksStatuses) Entry. >Thread-71744::DEBUG::2014-09-25 18:07:00,079::taskManager::123::Storage.TaskManager::(getAllTasksStatuses) Return: {} >Thread-71744::INFO::2014-09-25 18:07:00,079::logUtils::47::dispatcher::(wrapper) Run and protect: getAllTasksStatuses, Return response: {'allTasksStatus': {}} >Thread-71744::DEBUG::2014-09-25 18:07:00,079::task::1191::Storage.TaskManager.Task::(prepare) Task=`32dcecd4-8b73-4d05-8d5c-5ed5db76f4d1`::finished: {'allTasksStatus': {}} >Thread-71744::DEBUG::2014-09-25 18:07:00,079::task::595::Storage.TaskManager.Task::(_updateState) Task=`32dcecd4-8b73-4d05-8d5c-5ed5db76f4d1`::moving from state preparing -> state finished >Thread-71744::DEBUG::2014-09-25 18:07:00,080::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71744::DEBUG::2014-09-25 18:07:00,080::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71744::DEBUG::2014-09-25 18:07:00,080::task::993::Storage.TaskManager.Task::(_decref) Task=`32dcecd4-8b73-4d05-8d5c-5ed5db76f4d1`::ref 0 aborting False >Thread-71744::DEBUG::2014-09-25 18:07:00,080::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getAllTasksStatuses' in bridge with {} >Thread-71744::DEBUG::2014-09-25 18:07:00,081::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:07:00,104::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71745::DEBUG::2014-09-25 18:07:00,106::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:07:00,106::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71745::DEBUG::2014-09-25 18:07:00,108::task::595::Storage.TaskManager.Task::(_updateState) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::moving from state init -> state preparing >Thread-71745::INFO::2014-09-25 18:07:00,108::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71745::ERROR::2014-09-25 18:07:00,108::task::866::Storage.TaskManager.Task::(_setError) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71745::DEBUG::2014-09-25 18:07:00,109::task::885::Storage.TaskManager.Task::(_run) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::Task._run: 2cac493e-0448-4ace-8747-c7e9d4d2e8ed (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71745::DEBUG::2014-09-25 18:07:00,109::task::1217::Storage.TaskManager.Task::(stop) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::stopping in state preparing (force False) >Thread-71745::DEBUG::2014-09-25 18:07:00,109::task::993::Storage.TaskManager.Task::(_decref) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::ref 1 aborting True >Thread-71745::INFO::2014-09-25 18:07:00,109::task::1171::Storage.TaskManager.Task::(prepare) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::aborting: Task is aborted: u'' - code 100 >Thread-71745::DEBUG::2014-09-25 18:07:00,110::task::1176::Storage.TaskManager.Task::(prepare) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::Prepare: aborted: >Thread-71745::DEBUG::2014-09-25 18:07:00,110::task::993::Storage.TaskManager.Task::(_decref) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::ref 0 aborting True >Thread-71745::DEBUG::2014-09-25 18:07:00,110::task::928::Storage.TaskManager.Task::(_doAbort) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::Task._doAbort: force False >Thread-71745::DEBUG::2014-09-25 18:07:00,110::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71745::DEBUG::2014-09-25 18:07:00,110::task::595::Storage.TaskManager.Task::(_updateState) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::moving from state preparing -> state aborting >Thread-71745::DEBUG::2014-09-25 18:07:00,111::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::_aborting: recover policy none >Thread-71745::DEBUG::2014-09-25 18:07:00,111::task::595::Storage.TaskManager.Task::(_updateState) Task=`2cac493e-0448-4ace-8747-c7e9d4d2e8ed`::moving from state aborting -> state failed >Thread-71745::DEBUG::2014-09-25 18:07:00,111::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71745::DEBUG::2014-09-25 18:07:00,111::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71745::ERROR::2014-09-25 18:07:00,111::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71745::DEBUG::2014-09-25 18:07:00,112::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:07:01,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:01,240::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0208013 s, 49.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:07:01,526::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:07:01,528::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71746::DEBUG::2014-09-25 18:07:01,530::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:07:03,260::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:03,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205125 s, 49.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:03,600::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:03,600::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:03,600::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:03,600::__init__::232::IOProcess::(_processLogs) (110499) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:03,608::__init__::232::IOProcess::(_processLogs) (110499) Queuing response >Thread-79::DEBUG::2014-09-25 18:07:03,609::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:07:03,625::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0043318 s, 130 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:03,626::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:03,627::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:03,627::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:03,627::__init__::232::IOProcess::(_processLogs) (110500) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:03,631::__init__::232::IOProcess::(_processLogs) (110500) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:03,632::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:03,633::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:03,633::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:03,634::__init__::232::IOProcess::(_processLogs) (110501) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:03,634::__init__::232::IOProcess::(_processLogs) (110501) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:03,634::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:03,635::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:03,635::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:03,635::__init__::232::IOProcess::(_processLogs) (110502) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:03,636::__init__::232::IOProcess::(_processLogs) (110502) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:03,637::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:03,637::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:03,638::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:03,638::__init__::232::IOProcess::(_processLogs) (110503) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:03,638::__init__::232::IOProcess::(_processLogs) (110503) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:07:04,544::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:07:04,545::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71747::DEBUG::2014-09-25 18:07:04,548::task::595::Storage.TaskManager.Task::(_updateState) Task=`b4ef10e1-0bc4-416e-b911-8767c0710cf4`::moving from state init -> state preparing >Thread-71747::INFO::2014-09-25 18:07:04,548::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71747::INFO::2014-09-25 18:07:04,549::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0043318', 'lastCheck': '0.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '66.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00656396', 'lastCheck': '7.2', 'valid': True}} >Thread-71747::DEBUG::2014-09-25 18:07:04,549::task::1191::Storage.TaskManager.Task::(prepare) Task=`b4ef10e1-0bc4-416e-b911-8767c0710cf4`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0043318', 'lastCheck': '0.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '66.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00656396', 'lastCheck': '7.2', 'valid': True}} >Thread-71747::DEBUG::2014-09-25 18:07:04,549::task::595::Storage.TaskManager.Task::(_updateState) Task=`b4ef10e1-0bc4-416e-b911-8767c0710cf4`::moving from state preparing -> state finished >Thread-71747::DEBUG::2014-09-25 18:07:04,550::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71747::DEBUG::2014-09-25 18:07:04,550::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71747::DEBUG::2014-09-25 18:07:04,550::task::993::Storage.TaskManager.Task::(_decref) Task=`b4ef10e1-0bc4-416e-b911-8767c0710cf4`::ref 0 aborting False >Thread-71747::DEBUG::2014-09-25 18:07:04,552::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:07:04,565::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:07:04,566::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71748::DEBUG::2014-09-25 18:07:04,568::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:07:05,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:05,336::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013028 s, 78.6 MB/s\n'; <rc> = 0 >Thread-71701::WARNING::2014-09-25 18:07:07,043::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-29::DEBUG::2014-09-25 18:07:07,308::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:07,310::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:07,310::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:07,310::__init__::232::IOProcess::(_processLogs) (110504) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:07:07,311::__init__::232::IOProcess::(_processLogs) (110504) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:07,313::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:07,315::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:07,316::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:07,317::__init__::232::IOProcess::(_processLogs) (110505) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:07:07,319::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:07,320::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:07,322::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:07,323::__init__::232::IOProcess::(_processLogs) (110506) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:07:07,325::__init__::232::IOProcess::(_processLogs) (110506) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:07,325::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 18:07:07,327::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:07,327::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:07,327::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:07,328::__init__::232::IOProcess::(_processLogs) (110507) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 18:07:07,342::__init__::232::IOProcess::(_processLogs) (110507) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:07,343::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:07:07,344::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:07,344::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:07,344::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:07,344::__init__::232::IOProcess::(_processLogs) (110508) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 18:07:07,345::__init__::232::IOProcess::(_processLogs) (110508) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:07,345::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 18:07:07,347::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:07,347::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:07,348::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:07,348::__init__::232::IOProcess::(_processLogs) (110509) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:07,348::__init__::232::IOProcess::(_processLogs) (110509) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:07,349::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:07,349::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:07,349::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:07,350::__init__::232::IOProcess::(_processLogs) (110510) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:07:07,350::__init__::232::IOProcess::(_processLogs) (110510) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:07,350::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 18:07:07,351::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 18:07:07,351::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 18:07:07,352::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:07,352::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:07,353::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 18:07:07,354::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:07:07,354::__init__::232::IOProcess::(_processLogs) (110511) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:07,363::__init__::232::IOProcess::(_processLogs) (110511) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:07,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-81::DEBUG::2014-09-25 18:07:07,365::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000283808 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:07,374::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:07,375::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:07,375::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:07,375::__init__::232::IOProcess::(_processLogs) (110512) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:07,383::__init__::232::IOProcess::(_processLogs) (110512) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:07,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151003 s, 67.8 MB/s\n'; <rc> = 0 >Thread-71704::WARNING::2014-09-25 18:07:07,993::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:07:07,994::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:07:07,995::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Dummy-148::DEBUG::2014-09-25 18:07:09,409::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:09,891::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.468835 s, 2.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:07:10,150::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71752::DEBUG::2014-09-25 18:07:10,151::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:07:10,151::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71752::DEBUG::2014-09-25 18:07:10,153::task::595::Storage.TaskManager.Task::(_updateState) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::moving from state init -> state preparing >Thread-71752::INFO::2014-09-25 18:07:10,154::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71752::ERROR::2014-09-25 18:07:10,154::task::866::Storage.TaskManager.Task::(_setError) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71752::DEBUG::2014-09-25 18:07:10,155::task::885::Storage.TaskManager.Task::(_run) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::Task._run: c3fae562-315c-496d-b9d8-10aa340b5a42 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71752::DEBUG::2014-09-25 18:07:10,155::task::1217::Storage.TaskManager.Task::(stop) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::stopping in state preparing (force False) >Thread-71752::DEBUG::2014-09-25 18:07:10,155::task::993::Storage.TaskManager.Task::(_decref) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::ref 1 aborting True >Thread-71752::INFO::2014-09-25 18:07:10,155::task::1171::Storage.TaskManager.Task::(prepare) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::aborting: Task is aborted: u'' - code 100 >Thread-71752::DEBUG::2014-09-25 18:07:10,156::task::1176::Storage.TaskManager.Task::(prepare) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::Prepare: aborted: >Thread-71752::DEBUG::2014-09-25 18:07:10,156::task::993::Storage.TaskManager.Task::(_decref) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::ref 0 aborting True >Thread-71752::DEBUG::2014-09-25 18:07:10,156::task::928::Storage.TaskManager.Task::(_doAbort) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::Task._doAbort: force False >Thread-71752::DEBUG::2014-09-25 18:07:10,156::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71752::DEBUG::2014-09-25 18:07:10,156::task::595::Storage.TaskManager.Task::(_updateState) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::moving from state preparing -> state aborting >Thread-71752::DEBUG::2014-09-25 18:07:10,157::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::_aborting: recover policy none >Thread-71752::DEBUG::2014-09-25 18:07:10,157::task::595::Storage.TaskManager.Task::(_updateState) Task=`c3fae562-315c-496d-b9d8-10aa340b5a42`::moving from state aborting -> state failed >Thread-71752::DEBUG::2014-09-25 18:07:10,157::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71752::DEBUG::2014-09-25 18:07:10,157::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71752::ERROR::2014-09-25 18:07:10,157::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71752::DEBUG::2014-09-25 18:07:10,158::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:07:11,910::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:11,937::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015394 s, 66.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:13,639::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:13,639::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:13,639::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:13,639::__init__::232::IOProcess::(_processLogs) (110513) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:13,646::__init__::232::IOProcess::(_processLogs) (110513) Queuing response >Thread-79::DEBUG::2014-09-25 18:07:13,647::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:07:13,667::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00487733 s, 115 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:13,669::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:13,669::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:13,669::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:13,669::__init__::232::IOProcess::(_processLogs) (110514) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:13,673::__init__::232::IOProcess::(_processLogs) (110514) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:13,674::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:13,674::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:13,675::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:13,675::__init__::232::IOProcess::(_processLogs) (110515) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:13,676::__init__::232::IOProcess::(_processLogs) (110515) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:13,676::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:13,677::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:13,677::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:13,678::__init__::232::IOProcess::(_processLogs) (110516) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:13,678::__init__::232::IOProcess::(_processLogs) (110516) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:13,678::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:13,679::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:13,680::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:13,680::__init__::232::IOProcess::(_processLogs) (110517) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:13,680::__init__::232::IOProcess::(_processLogs) (110517) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:13,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:13,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013441 s, 76.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:07:16,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:16,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0221832 s, 46.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:17,387::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:17,387::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:17,388::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:17,388::__init__::232::IOProcess::(_processLogs) (110518) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:17,392::__init__::232::IOProcess::(_processLogs) (110518) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:17,393::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:07:17,409::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00679253 s, 54.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:17,410::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:17,411::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:17,411::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:17,411::__init__::232::IOProcess::(_processLogs) (110519) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:17,416::__init__::232::IOProcess::(_processLogs) (110519) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:18,000::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:18,001::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:18,001::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:18,001::__init__::232::IOProcess::(_processLogs) (110520) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:07:18,002::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:18,002::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:18,002::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:18,002::__init__::232::IOProcess::(_processLogs) (110521) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:07:18,003::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:18,005::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:18,005::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:18,005::__init__::232::IOProcess::(_processLogs) (110522) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:07:18,006::__init__::232::IOProcess::(_processLogs) (110520) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:18,006::__init__::232::IOProcess::(_processLogs) (110522) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:18,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:18,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0217073 s, 47.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:07:20,197::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71756::DEBUG::2014-09-25 18:07:20,198::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:07:20,198::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71756::DEBUG::2014-09-25 18:07:20,201::task::595::Storage.TaskManager.Task::(_updateState) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::moving from state init -> state preparing >Thread-71756::INFO::2014-09-25 18:07:20,201::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71756::ERROR::2014-09-25 18:07:20,201::task::866::Storage.TaskManager.Task::(_setError) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71756::DEBUG::2014-09-25 18:07:20,202::task::885::Storage.TaskManager.Task::(_run) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::Task._run: 9d5aa9dd-6ab6-4784-bb35-0ce4488859a3 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71756::DEBUG::2014-09-25 18:07:20,202::task::1217::Storage.TaskManager.Task::(stop) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::stopping in state preparing (force False) >Thread-71756::DEBUG::2014-09-25 18:07:20,202::task::993::Storage.TaskManager.Task::(_decref) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::ref 1 aborting True >Thread-71756::INFO::2014-09-25 18:07:20,202::task::1171::Storage.TaskManager.Task::(prepare) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::aborting: Task is aborted: u'' - code 100 >Thread-71756::DEBUG::2014-09-25 18:07:20,203::task::1176::Storage.TaskManager.Task::(prepare) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::Prepare: aborted: >Thread-71756::DEBUG::2014-09-25 18:07:20,203::task::993::Storage.TaskManager.Task::(_decref) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::ref 0 aborting True >Thread-71756::DEBUG::2014-09-25 18:07:20,203::task::928::Storage.TaskManager.Task::(_doAbort) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::Task._doAbort: force False >Thread-71756::DEBUG::2014-09-25 18:07:20,203::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71756::DEBUG::2014-09-25 18:07:20,204::task::595::Storage.TaskManager.Task::(_updateState) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::moving from state preparing -> state aborting >Thread-71756::DEBUG::2014-09-25 18:07:20,204::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::_aborting: recover policy none >Thread-71756::DEBUG::2014-09-25 18:07:20,204::task::595::Storage.TaskManager.Task::(_updateState) Task=`9d5aa9dd-6ab6-4784-bb35-0ce4488859a3`::moving from state aborting -> state failed >Thread-71756::DEBUG::2014-09-25 18:07:20,204::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71756::DEBUG::2014-09-25 18:07:20,204::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71756::ERROR::2014-09-25 18:07:20,205::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71756::DEBUG::2014-09-25 18:07:20,205::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:07:20,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:20,307::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148778 s, 68.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:07:22,327::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:22,363::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0204566 s, 50.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:23,681::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:23,681::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:23,681::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:23,681::__init__::232::IOProcess::(_processLogs) (110523) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:23,689::__init__::232::IOProcess::(_processLogs) (110523) Queuing response >Thread-79::DEBUG::2014-09-25 18:07:23,691::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:07:23,712::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00388466 s, 145 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:23,713::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:23,713::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:23,714::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:23,714::__init__::232::IOProcess::(_processLogs) (110524) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:23,718::__init__::232::IOProcess::(_processLogs) (110524) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:23,720::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:23,720::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:23,721::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:23,721::__init__::232::IOProcess::(_processLogs) (110525) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:23,721::__init__::232::IOProcess::(_processLogs) (110525) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:23,722::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:23,722::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:23,723::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:23,723::__init__::232::IOProcess::(_processLogs) (110526) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:23,724::__init__::232::IOProcess::(_processLogs) (110526) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:23,724::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:23,724::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:23,725::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:23,726::__init__::232::IOProcess::(_processLogs) (110527) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:23,726::__init__::232::IOProcess::(_processLogs) (110527) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:24,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:24,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183582 s, 55.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:07:26,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:26,471::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172883 s, 59.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:27,418::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:27,419::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:27,419::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:27,419::__init__::232::IOProcess::(_processLogs) (110528) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:27,420::__init__::232::IOProcess::(_processLogs) (110528) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:27,421::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:07:27,430::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000320056 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:27,431::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:27,432::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:27,432::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:27,432::__init__::232::IOProcess::(_processLogs) (110529) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:27,433::__init__::232::IOProcess::(_processLogs) (110529) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:28,490::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:28,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155765 s, 65.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:07:30,234::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:07:30,235::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71757::DEBUG::2014-09-25 18:07:30,235::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71757::DEBUG::2014-09-25 18:07:30,237::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::moving from state init -> state preparing >Thread-71757::INFO::2014-09-25 18:07:30,237::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71757::ERROR::2014-09-25 18:07:30,238::task::866::Storage.TaskManager.Task::(_setError) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71757::DEBUG::2014-09-25 18:07:30,238::task::885::Storage.TaskManager.Task::(_run) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::Task._run: 0b737da1-c9e9-4ab7-a15c-b71980598267 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71757::DEBUG::2014-09-25 18:07:30,239::task::1217::Storage.TaskManager.Task::(stop) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::stopping in state preparing (force False) >Thread-71757::DEBUG::2014-09-25 18:07:30,239::task::993::Storage.TaskManager.Task::(_decref) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::ref 1 aborting True >Thread-71757::INFO::2014-09-25 18:07:30,239::task::1171::Storage.TaskManager.Task::(prepare) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::aborting: Task is aborted: u'' - code 100 >Thread-71757::DEBUG::2014-09-25 18:07:30,239::task::1176::Storage.TaskManager.Task::(prepare) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::Prepare: aborted: >Thread-71757::DEBUG::2014-09-25 18:07:30,239::task::993::Storage.TaskManager.Task::(_decref) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::ref 0 aborting True >Thread-71757::DEBUG::2014-09-25 18:07:30,240::task::928::Storage.TaskManager.Task::(_doAbort) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::Task._doAbort: force False >Thread-71757::DEBUG::2014-09-25 18:07:30,240::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71757::DEBUG::2014-09-25 18:07:30,240::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::moving from state preparing -> state aborting >Thread-71757::DEBUG::2014-09-25 18:07:30,240::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::_aborting: recover policy none >Thread-71757::DEBUG::2014-09-25 18:07:30,241::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b737da1-c9e9-4ab7-a15c-b71980598267`::moving from state aborting -> state failed >Thread-71757::DEBUG::2014-09-25 18:07:30,241::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71757::DEBUG::2014-09-25 18:07:30,241::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71757::ERROR::2014-09-25 18:07:30,241::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71757::DEBUG::2014-09-25 18:07:30,242::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:07:30,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:30,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0210818 s, 48.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:07:32,596::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:32,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014829 s, 69.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:33,727::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:33,727::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:33,727::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:33,728::__init__::232::IOProcess::(_processLogs) (110530) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:33,731::__init__::232::IOProcess::(_processLogs) (110530) Queuing response >Thread-79::DEBUG::2014-09-25 18:07:33,732::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:07:33,749::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00643731 s, 87.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:33,750::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:33,750::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:33,751::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:33,751::__init__::232::IOProcess::(_processLogs) (110531) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:33,754::__init__::232::IOProcess::(_processLogs) (110531) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:33,756::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:33,756::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:33,757::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:33,757::__init__::232::IOProcess::(_processLogs) (110532) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:33,757::__init__::232::IOProcess::(_processLogs) (110532) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:33,758::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:33,758::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:33,759::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:33,759::__init__::232::IOProcess::(_processLogs) (110533) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:33,760::__init__::232::IOProcess::(_processLogs) (110533) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:33,760::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:33,760::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:33,761::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:33,762::__init__::232::IOProcess::(_processLogs) (110534) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:33,762::__init__::232::IOProcess::(_processLogs) (110534) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:34,644::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:34,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131103 s, 78.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:07:36,686::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:36,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.260751 s, 3.9 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 18:07:37,435::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 18:07:37,438::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:37,439::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:37,440::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:37,440::__init__::232::IOProcess::(_processLogs) (110535) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:07:37,440::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:37,441::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:37,441::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:37,441::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:37,441::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:37,442::__init__::232::IOProcess::(_processLogs) (110536) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:07:37,442::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:37,442::__init__::232::IOProcess::(_processLogs) (110537) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:07:37,443::__init__::232::IOProcess::(_processLogs) (110537) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:37,443::__init__::232::IOProcess::(_processLogs) (110535) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:37,447::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 18:07:37,448::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:37,448::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:37,448::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:37,449::__init__::232::IOProcess::(_processLogs) (110538) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 18:07:37,465::__init__::232::IOProcess::(_processLogs) (110538) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:37,465::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:07:37,467::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:37,467::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:37,467::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:37,468::__init__::232::IOProcess::(_processLogs) (110539) Got request for method 'readfile' >Thread-81::DEBUG::2014-09-25 18:07:37,468::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 18:07:37,468::__init__::232::IOProcess::(_processLogs) (110539) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:37,470::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:37,471::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:37,471::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:37,471::__init__::232::IOProcess::(_processLogs) (110540) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:37,472::__init__::232::IOProcess::(_processLogs) (110540) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:37,472::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:37,473::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-81::DEBUG::2014-09-25 18:07:37,473::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 18:07:37,473::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::INFO::2014-09-25 18:07:37,474::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-29::DEBUG::2014-09-25 18:07:37,475::__init__::232::IOProcess::(_processLogs) (110541) Got request for method 'glob' >Thread-81::INFO::2014-09-25 18:07:37,475::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 18:07:37,476::__init__::232::IOProcess::(_processLogs) (110541) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:37,477::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:37,477::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:37,478::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-81::DEBUG::2014-09-25 18:07:37,479::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:07:37,479::__init__::232::IOProcess::(_processLogs) (110542) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:37,488::__init__::232::IOProcess::(_processLogs) (110542) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:37,489::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000456184 s, 813 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:37,490::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:37,490::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:37,491::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:37,492::__init__::232::IOProcess::(_processLogs) (110543) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:37,492::__init__::232::IOProcess::(_processLogs) (110543) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:38,977::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:39,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0231698 s, 44.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:07:40,264::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:07:40,266::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71761::DEBUG::2014-09-25 18:07:40,266::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71761::DEBUG::2014-09-25 18:07:40,268::task::595::Storage.TaskManager.Task::(_updateState) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::moving from state init -> state preparing >Thread-71761::INFO::2014-09-25 18:07:40,268::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71761::ERROR::2014-09-25 18:07:40,269::task::866::Storage.TaskManager.Task::(_setError) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71761::DEBUG::2014-09-25 18:07:40,269::task::885::Storage.TaskManager.Task::(_run) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::Task._run: 159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71761::DEBUG::2014-09-25 18:07:40,270::task::1217::Storage.TaskManager.Task::(stop) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::stopping in state preparing (force False) >Thread-71761::DEBUG::2014-09-25 18:07:40,270::task::993::Storage.TaskManager.Task::(_decref) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::ref 1 aborting True >Thread-71761::INFO::2014-09-25 18:07:40,270::task::1171::Storage.TaskManager.Task::(prepare) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::aborting: Task is aborted: u'' - code 100 >Thread-71761::DEBUG::2014-09-25 18:07:40,270::task::1176::Storage.TaskManager.Task::(prepare) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::Prepare: aborted: >Thread-71761::DEBUG::2014-09-25 18:07:40,271::task::993::Storage.TaskManager.Task::(_decref) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::ref 0 aborting True >Thread-71761::DEBUG::2014-09-25 18:07:40,271::task::928::Storage.TaskManager.Task::(_doAbort) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::Task._doAbort: force False >Thread-71761::DEBUG::2014-09-25 18:07:40,271::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71761::DEBUG::2014-09-25 18:07:40,271::task::595::Storage.TaskManager.Task::(_updateState) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::moving from state preparing -> state aborting >Thread-71761::DEBUG::2014-09-25 18:07:40,271::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::_aborting: recover policy none >Thread-71761::DEBUG::2014-09-25 18:07:40,272::task::595::Storage.TaskManager.Task::(_updateState) Task=`159ce626-d4ec-4ca5-9e1e-af4dbc7e7ab0`::moving from state aborting -> state failed >Thread-71761::DEBUG::2014-09-25 18:07:40,272::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71761::DEBUG::2014-09-25 18:07:40,272::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71761::ERROR::2014-09-25 18:07:40,272::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71761::DEBUG::2014-09-25 18:07:40,273::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:07:41,032::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:41,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136332 s, 75.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:07:43,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:43,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153456 s, 66.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:43,763::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:43,763::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:43,763::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:43,764::__init__::232::IOProcess::(_processLogs) (110544) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:43,766::__init__::232::IOProcess::(_processLogs) (110544) Queuing response >Thread-79::DEBUG::2014-09-25 18:07:43,767::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:07:43,778::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00068687 s, 818 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:43,779::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:43,780::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:43,780::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:43,780::__init__::232::IOProcess::(_processLogs) (110545) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:43,783::__init__::232::IOProcess::(_processLogs) (110545) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:43,785::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:43,785::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:43,786::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:43,786::__init__::232::IOProcess::(_processLogs) (110546) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:43,787::__init__::232::IOProcess::(_processLogs) (110546) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:43,787::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:43,787::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:43,788::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:43,789::__init__::232::IOProcess::(_processLogs) (110547) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:43,789::__init__::232::IOProcess::(_processLogs) (110547) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:43,789::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:43,790::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:43,790::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:43,791::__init__::232::IOProcess::(_processLogs) (110548) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:43,791::__init__::232::IOProcess::(_processLogs) (110548) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:45,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:45,161::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0231496 s, 44.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:07:47,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:47,200::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0120988 s, 84.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:47,493::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:47,493::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:47,493::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:47,493::__init__::232::IOProcess::(_processLogs) (110549) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:47,495::__init__::232::IOProcess::(_processLogs) (110549) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:47,496::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:07:47,505::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00276726 s, 134 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:47,506::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:47,506::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:47,507::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:47,507::__init__::232::IOProcess::(_processLogs) (110550) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:47,512::__init__::232::IOProcess::(_processLogs) (110550) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:49,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:49,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0258871 s, 39.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:07:50,296::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71762::DEBUG::2014-09-25 18:07:50,297::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:07:50,297::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71762::DEBUG::2014-09-25 18:07:50,300::task::595::Storage.TaskManager.Task::(_updateState) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::moving from state init -> state preparing >Thread-71762::INFO::2014-09-25 18:07:50,300::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71762::ERROR::2014-09-25 18:07:50,300::task::866::Storage.TaskManager.Task::(_setError) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71762::DEBUG::2014-09-25 18:07:50,301::task::885::Storage.TaskManager.Task::(_run) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::Task._run: b48e3e07-b61a-418a-a18d-fb8393ae3433 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71762::DEBUG::2014-09-25 18:07:50,301::task::1217::Storage.TaskManager.Task::(stop) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::stopping in state preparing (force False) >Thread-71762::DEBUG::2014-09-25 18:07:50,301::task::993::Storage.TaskManager.Task::(_decref) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::ref 1 aborting True >Thread-71762::INFO::2014-09-25 18:07:50,302::task::1171::Storage.TaskManager.Task::(prepare) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::aborting: Task is aborted: u'' - code 100 >Thread-71762::DEBUG::2014-09-25 18:07:50,302::task::1176::Storage.TaskManager.Task::(prepare) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::Prepare: aborted: >Thread-71762::DEBUG::2014-09-25 18:07:50,302::task::993::Storage.TaskManager.Task::(_decref) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::ref 0 aborting True >Thread-71762::DEBUG::2014-09-25 18:07:50,302::task::928::Storage.TaskManager.Task::(_doAbort) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::Task._doAbort: force False >Thread-71762::DEBUG::2014-09-25 18:07:50,303::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71762::DEBUG::2014-09-25 18:07:50,303::task::595::Storage.TaskManager.Task::(_updateState) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::moving from state preparing -> state aborting >Thread-71762::DEBUG::2014-09-25 18:07:50,303::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::_aborting: recover policy none >Thread-71762::DEBUG::2014-09-25 18:07:50,303::task::595::Storage.TaskManager.Task::(_updateState) Task=`b48e3e07-b61a-418a-a18d-fb8393ae3433`::moving from state aborting -> state failed >Thread-71762::DEBUG::2014-09-25 18:07:50,303::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71762::DEBUG::2014-09-25 18:07:50,304::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71762::ERROR::2014-09-25 18:07:50,304::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71762::DEBUG::2014-09-25 18:07:50,304::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:07:51,272::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:51,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013227 s, 77.4 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:07:53,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:53,338::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133198 s, 76.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:53,792::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:53,792::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:53,792::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:53,793::__init__::232::IOProcess::(_processLogs) (110551) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:53,800::__init__::232::IOProcess::(_processLogs) (110551) Queuing response >Thread-79::DEBUG::2014-09-25 18:07:53,802::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:07:53,826::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00945396 s, 59.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:53,827::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:53,827::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:53,828::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:53,828::__init__::232::IOProcess::(_processLogs) (110552) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:53,833::__init__::232::IOProcess::(_processLogs) (110552) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:53,835::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:53,835::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:53,835::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:53,835::__init__::232::IOProcess::(_processLogs) (110553) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:53,841::__init__::232::IOProcess::(_processLogs) (110553) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:53,842::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:53,842::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:53,843::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:53,843::__init__::232::IOProcess::(_processLogs) (110554) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:53,851::__init__::232::IOProcess::(_processLogs) (110554) Queuing response >Thread-29::DEBUG::2014-09-25 18:07:53,852::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:53,852::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:53,852::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:53,853::__init__::232::IOProcess::(_processLogs) (110555) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:07:53,859::__init__::232::IOProcess::(_processLogs) (110555) Queuing response >Dummy-148::DEBUG::2014-09-25 18:07:55,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:55,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162222 s, 63.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:07:57,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:57,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.025256 s, 40.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:57,514::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:57,514::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:57,515::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:57,515::__init__::232::IOProcess::(_processLogs) (110556) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:57,519::__init__::232::IOProcess::(_processLogs) (110556) Queuing response >Thread-81::DEBUG::2014-09-25 18:07:57,520::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:07:57,533::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00304353 s, 122 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:07:57,534::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:07:57,534::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:07:57,534::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:07:57,535::__init__::232::IOProcess::(_processLogs) (110557) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:07:57,536::__init__::232::IOProcess::(_processLogs) (110557) Queuing response >Thread-71738::WARNING::2014-09-25 18:07:58,913::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-71741::WARNING::2014-09-25 18:07:59,043::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-71736::INFO::2014-09-25 18:07:59,044::sp::1120::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f to /rhev/data-center/00000002-0002-0002-0002-0000000001ad/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-71736::DEBUG::2014-09-25 18:07:59,044::sp::1131::Storage.StoragePool::(_linkStorageDomain) link already present skipping creation for /rhev/data-center/00000002-0002-0002-0002-0000000001ad/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-71736::INFO::2014-09-25 18:07:59,044::sp::1120::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc to /rhev/data-center/00000002-0002-0002-0002-0000000001ad/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-71736::DEBUG::2014-09-25 18:07:59,045::sp::1131::Storage.StoragePool::(_linkStorageDomain) link already present skipping creation for /rhev/data-center/00000002-0002-0002-0002-0000000001ad/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-71736::INFO::2014-09-25 18:07:59,045::sp::1120::Storage.StoragePool::(_linkStorageDomain) Linking /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc to /rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd >Thread-71736::DEBUG::2014-09-25 18:07:59,045::sp::1131::Storage.StoragePool::(_linkStorageDomain) link already present skipping creation for /rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd >Thread-71736::DEBUG::2014-09-25 18:07:59,045::sp::1221::Storage.StoragePool::(__rebuild) collecting stale storage domain link /rhev/data-center/00000002-0002-0002-0002-0000000001ad/fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-71736::DEBUG::2014-09-25 18:07:59,046::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.00000002-0002-0002-0002-0000000001ad' >Thread-71736::DEBUG::2014-09-25 18:07:59,046::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.00000002-0002-0002-0002-0000000001ad' (0 active users) >Thread-71736::DEBUG::2014-09-25 18:07:59,047::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.00000002-0002-0002-0002-0000000001ad' is free, finding out if anyone is waiting for it. >Thread-71736::DEBUG::2014-09-25 18:07:59,047::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.00000002-0002-0002-0002-0000000001ad', Clearing records. >Thread-71736::DEBUG::2014-09-25 18:07:59,047::resourceManager::616::Storage.ResourceManager::(releaseResource) Trying to release resource 'Storage.HsmDomainMonitorLock' >Thread-71736::DEBUG::2014-09-25 18:07:59,048::resourceManager::635::Storage.ResourceManager::(releaseResource) Released resource 'Storage.HsmDomainMonitorLock' (0 active users) >Thread-71736::DEBUG::2014-09-25 18:07:59,048::resourceManager::641::Storage.ResourceManager::(releaseResource) Resource 'Storage.HsmDomainMonitorLock' is free, finding out if anyone is waiting for it. >Thread-71736::DEBUG::2014-09-25 18:07:59,048::resourceManager::649::Storage.ResourceManager::(releaseResource) No one is waiting for resource 'Storage.HsmDomainMonitorLock', Clearing records. >Thread-71736::INFO::2014-09-25 18:07:59,048::logUtils::47::dispatcher::(wrapper) Run and protect: connectStoragePool, Return response: True >Thread-71736::DEBUG::2014-09-25 18:07:59,049::task::1191::Storage.TaskManager.Task::(prepare) Task=`e73af8b8-c6af-4271-83d8-95e8fcc6bd90`::finished: True >Thread-71736::DEBUG::2014-09-25 18:07:59,049::task::595::Storage.TaskManager.Task::(_updateState) Task=`e73af8b8-c6af-4271-83d8-95e8fcc6bd90`::moving from state preparing -> state finished >Thread-71736::DEBUG::2014-09-25 18:07:59,049::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71736::DEBUG::2014-09-25 18:07:59,049::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71736::DEBUG::2014-09-25 18:07:59,050::task::993::Storage.TaskManager.Task::(_decref) Task=`e73af8b8-c6af-4271-83d8-95e8fcc6bd90`::ref 0 aborting False >Thread-71736::DEBUG::2014-09-25 18:07:59,050::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.connect' in bridge with True >Thread-71736::DEBUG::2014-09-25 18:07:59,050::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:07:59,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:07:59,484::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013057 s, 78.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:00,328::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71763::DEBUG::2014-09-25 18:08:00,329::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:08:00,329::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71763::DEBUG::2014-09-25 18:08:00,331::task::595::Storage.TaskManager.Task::(_updateState) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::moving from state init -> state preparing >Thread-71763::INFO::2014-09-25 18:08:00,332::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71763::ERROR::2014-09-25 18:08:00,332::task::866::Storage.TaskManager.Task::(_setError) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71763::DEBUG::2014-09-25 18:08:00,333::task::885::Storage.TaskManager.Task::(_run) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::Task._run: 0624c669-35b9-4d6a-bd26-e51a1c91a5d4 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71763::DEBUG::2014-09-25 18:08:00,333::task::1217::Storage.TaskManager.Task::(stop) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::stopping in state preparing (force False) >Thread-71763::DEBUG::2014-09-25 18:08:00,333::task::993::Storage.TaskManager.Task::(_decref) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::ref 1 aborting True >Thread-71763::INFO::2014-09-25 18:08:00,333::task::1171::Storage.TaskManager.Task::(prepare) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::aborting: Task is aborted: u'' - code 100 >Thread-71763::DEBUG::2014-09-25 18:08:00,334::task::1176::Storage.TaskManager.Task::(prepare) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::Prepare: aborted: >Thread-71763::DEBUG::2014-09-25 18:08:00,334::task::993::Storage.TaskManager.Task::(_decref) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::ref 0 aborting True >Thread-71763::DEBUG::2014-09-25 18:08:00,334::task::928::Storage.TaskManager.Task::(_doAbort) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::Task._doAbort: force False >Thread-71763::DEBUG::2014-09-25 18:08:00,334::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71763::DEBUG::2014-09-25 18:08:00,335::task::595::Storage.TaskManager.Task::(_updateState) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::moving from state preparing -> state aborting >Thread-71763::DEBUG::2014-09-25 18:08:00,335::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::_aborting: recover policy none >Thread-71763::DEBUG::2014-09-25 18:08:00,335::task::595::Storage.TaskManager.Task::(_updateState) Task=`0624c669-35b9-4d6a-bd26-e51a1c91a5d4`::moving from state aborting -> state failed >Thread-71763::DEBUG::2014-09-25 18:08:00,335::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71763::DEBUG::2014-09-25 18:08:00,335::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71763::ERROR::2014-09-25 18:08:00,336::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71763::DEBUG::2014-09-25 18:08:00,336::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:01,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:01,530::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148651 s, 68.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:02,063::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:02,064::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71764::DEBUG::2014-09-25 18:08:02,066::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:03,550::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:03,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201608 s, 50.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:03,861::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:03,861::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:03,861::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:03,862::__init__::232::IOProcess::(_processLogs) (110558) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:03,868::__init__::232::IOProcess::(_processLogs) (110558) Queuing response >Thread-79::DEBUG::2014-09-25 18:08:03,868::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:08:03,881::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00394443 s, 142 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:03,882::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:03,882::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:03,883::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:03,883::__init__::232::IOProcess::(_processLogs) (110559) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:03,887::__init__::232::IOProcess::(_processLogs) (110559) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:03,889::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:03,889::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:03,890::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:03,890::__init__::232::IOProcess::(_processLogs) (110560) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:03,890::__init__::232::IOProcess::(_processLogs) (110560) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:03,891::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:03,891::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:03,892::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:03,892::__init__::232::IOProcess::(_processLogs) (110561) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:03,892::__init__::232::IOProcess::(_processLogs) (110561) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:03,893::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:03,893::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:03,894::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:03,894::__init__::232::IOProcess::(_processLogs) (110562) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:03,895::__init__::232::IOProcess::(_processLogs) (110562) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:05,082::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:05,083::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71765::DEBUG::2014-09-25 18:08:05,085::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:05,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:05,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0258288 s, 39.6 MB/s\n'; <rc> = 0 >Thread-71750::WARNING::2014-09-25 18:08:07,309::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-29::DEBUG::2014-09-25 18:08:07,538::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:07,538::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:07,539::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:07,539::__init__::232::IOProcess::(_processLogs) (110563) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:07,544::__init__::232::IOProcess::(_processLogs) (110563) Queuing response >Thread-81::DEBUG::2014-09-25 18:08:07,545::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:08:07,562::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00168433 s, 220 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:07,563::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:07,563::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:07,564::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:07,564::__init__::232::IOProcess::(_processLogs) (110564) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:07,565::__init__::232::IOProcess::(_processLogs) (110564) Queuing response >Dummy-148::DEBUG::2014-09-25 18:08:07,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:07,897::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183295 s, 55.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:08,099::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:08,101::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71766::DEBUG::2014-09-25 18:08:08,103::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:09,916::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:09,945::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161238 s, 63.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:10,361::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71767::DEBUG::2014-09-25 18:08:10,363::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:08:10,363::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71767::DEBUG::2014-09-25 18:08:10,365::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::moving from state init -> state preparing >Thread-71767::INFO::2014-09-25 18:08:10,366::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71767::ERROR::2014-09-25 18:08:10,366::task::866::Storage.TaskManager.Task::(_setError) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71767::DEBUG::2014-09-25 18:08:10,367::task::885::Storage.TaskManager.Task::(_run) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::Task._run: 3a9da3ae-8d32-4680-b3fd-2fbb94c9badd (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71767::DEBUG::2014-09-25 18:08:10,367::task::1217::Storage.TaskManager.Task::(stop) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::stopping in state preparing (force False) >Thread-71767::DEBUG::2014-09-25 18:08:10,367::task::993::Storage.TaskManager.Task::(_decref) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::ref 1 aborting True >Thread-71767::INFO::2014-09-25 18:08:10,368::task::1171::Storage.TaskManager.Task::(prepare) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::aborting: Task is aborted: u'' - code 100 >Thread-71767::DEBUG::2014-09-25 18:08:10,368::task::1176::Storage.TaskManager.Task::(prepare) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::Prepare: aborted: >Thread-71767::DEBUG::2014-09-25 18:08:10,368::task::993::Storage.TaskManager.Task::(_decref) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::ref 0 aborting True >Thread-71767::DEBUG::2014-09-25 18:08:10,368::task::928::Storage.TaskManager.Task::(_doAbort) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::Task._doAbort: force False >Thread-71767::DEBUG::2014-09-25 18:08:10,368::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71767::DEBUG::2014-09-25 18:08:10,369::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::moving from state preparing -> state aborting >Thread-71767::DEBUG::2014-09-25 18:08:10,369::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::_aborting: recover policy none >Thread-71767::DEBUG::2014-09-25 18:08:10,369::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a9da3ae-8d32-4680-b3fd-2fbb94c9badd`::moving from state aborting -> state failed >Thread-71767::DEBUG::2014-09-25 18:08:10,369::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71767::DEBUG::2014-09-25 18:08:10,369::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71767::ERROR::2014-09-25 18:08:10,370::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71767::DEBUG::2014-09-25 18:08:10,370::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:11,120::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:11,121::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71768::DEBUG::2014-09-25 18:08:11,123::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:11,965::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:12,002::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0206282 s, 49.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:13,895::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:13,895::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:13,896::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:13,896::__init__::232::IOProcess::(_processLogs) (110565) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:13,900::__init__::232::IOProcess::(_processLogs) (110565) Queuing response >Thread-79::DEBUG::2014-09-25 18:08:13,901::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:08:13,916::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00288191 s, 195 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:13,917::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:13,918::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:13,918::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:13,918::__init__::232::IOProcess::(_processLogs) (110566) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:13,920::__init__::232::IOProcess::(_processLogs) (110566) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:13,921::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:13,921::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:13,922::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:13,922::__init__::232::IOProcess::(_processLogs) (110567) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:13,922::__init__::232::IOProcess::(_processLogs) (110567) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:13,923::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:13,924::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:13,924::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:13,924::__init__::232::IOProcess::(_processLogs) (110568) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:13,925::__init__::232::IOProcess::(_processLogs) (110568) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:13,925::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:13,925::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:13,926::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:13,927::__init__::232::IOProcess::(_processLogs) (110569) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:13,927::__init__::232::IOProcess::(_processLogs) (110569) Queuing response >Dummy-148::DEBUG::2014-09-25 18:08:14,025::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:14,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192411 s, 53.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:14,139::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:14,140::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71769::DEBUG::2014-09-25 18:08:14,143::task::595::Storage.TaskManager.Task::(_updateState) Task=`158c9573-c669-491e-a422-e54d86ba4762`::moving from state init -> state preparing >Thread-71769::INFO::2014-09-25 18:08:14,144::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71769::INFO::2014-09-25 18:08:14,144::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00288191', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '66.1', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00168433', 'lastCheck': '6.6', 'valid': True}} >Thread-71769::DEBUG::2014-09-25 18:08:14,144::task::1191::Storage.TaskManager.Task::(prepare) Task=`158c9573-c669-491e-a422-e54d86ba4762`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00288191', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '66.1', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00168433', 'lastCheck': '6.6', 'valid': True}} >Thread-71769::DEBUG::2014-09-25 18:08:14,145::task::595::Storage.TaskManager.Task::(_updateState) Task=`158c9573-c669-491e-a422-e54d86ba4762`::moving from state preparing -> state finished >Thread-71769::DEBUG::2014-09-25 18:08:14,145::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71769::DEBUG::2014-09-25 18:08:14,145::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71769::DEBUG::2014-09-25 18:08:14,145::task::993::Storage.TaskManager.Task::(_decref) Task=`158c9573-c669-491e-a422-e54d86ba4762`::ref 0 aborting False >Thread-71769::DEBUG::2014-09-25 18:08:14,148::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:14,163::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:14,164::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71770::DEBUG::2014-09-25 18:08:14,166::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:16,076::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:16,113::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0253299 s, 40.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:17,483::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:17,485::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71771::DEBUG::2014-09-25 18:08:17,487::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:08:17,568::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:17,568::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:17,568::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:17,568::__init__::232::IOProcess::(_processLogs) (110570) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:17,570::__init__::232::IOProcess::(_processLogs) (110570) Queuing response >Thread-81::DEBUG::2014-09-25 18:08:17,572::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:08:17,589::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00492005 s, 75.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:17,591::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:17,591::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:17,591::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:17,591::__init__::232::IOProcess::(_processLogs) (110571) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:17,593::__init__::232::IOProcess::(_processLogs) (110571) Queuing response >Thread-71754::WARNING::2014-09-25 18:08:17,999::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:08:17,999::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:08:18,000::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Dummy-148::DEBUG::2014-09-25 18:08:18,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:18,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0219064 s, 46.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:08:20,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:20,395::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71772::DEBUG::2014-09-25 18:08:20,399::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:08:20,399::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71772::DEBUG::2014-09-25 18:08:20,405::task::595::Storage.TaskManager.Task::(_updateState) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::moving from state init -> state preparing >Thread-71772::INFO::2014-09-25 18:08:20,406::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71772::ERROR::2014-09-25 18:08:20,408::task::866::Storage.TaskManager.Task::(_setError) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71772::DEBUG::2014-09-25 18:08:20,410::task::885::Storage.TaskManager.Task::(_run) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::Task._run: 5fdf0aa7-0aa7-4042-bba0-fc899af245c1 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71772::DEBUG::2014-09-25 18:08:20,411::task::1217::Storage.TaskManager.Task::(stop) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::stopping in state preparing (force False) >Thread-71772::DEBUG::2014-09-25 18:08:20,412::task::993::Storage.TaskManager.Task::(_decref) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::ref 1 aborting True >Thread-71772::INFO::2014-09-25 18:08:20,413::task::1171::Storage.TaskManager.Task::(prepare) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::aborting: Task is aborted: u'' - code 100 >Thread-71772::DEBUG::2014-09-25 18:08:20,414::task::1176::Storage.TaskManager.Task::(prepare) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::Prepare: aborted: >Thread-71772::DEBUG::2014-09-25 18:08:20,415::task::993::Storage.TaskManager.Task::(_decref) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::ref 0 aborting True >Thread-71772::DEBUG::2014-09-25 18:08:20,416::task::928::Storage.TaskManager.Task::(_doAbort) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::Task._doAbort: force False >Thread-71772::DEBUG::2014-09-25 18:08:20,417::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71772::DEBUG::2014-09-25 18:08:20,418::task::595::Storage.TaskManager.Task::(_updateState) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::moving from state preparing -> state aborting >Thread-71772::DEBUG::2014-09-25 18:08:20,419::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::_aborting: recover policy none >Thread-71772::DEBUG::2014-09-25 18:08:20,420::task::595::Storage.TaskManager.Task::(_updateState) Task=`5fdf0aa7-0aa7-4042-bba0-fc899af245c1`::moving from state aborting -> state failed >Thread-71772::DEBUG::2014-09-25 18:08:20,421::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71772::DEBUG::2014-09-25 18:08:20,421::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71772::ERROR::2014-09-25 18:08:20,423::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Dummy-148::DEBUG::2014-09-25 18:08:20,423::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0426513 s, 24.0 MB/s\n'; <rc> = 0 >Thread-71772::DEBUG::2014-09-25 18:08:20,425::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:20,503::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:20,503::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71773::DEBUG::2014-09-25 18:08:20,505::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:22,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:22,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170978 s, 59.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:23,520::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:23,521::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71774::DEBUG::2014-09-25 18:08:23,523::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:08:23,927::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:23,927::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:23,928::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:23,928::__init__::232::IOProcess::(_processLogs) (110572) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:23,936::__init__::232::IOProcess::(_processLogs) (110572) Queuing response >Thread-79::DEBUG::2014-09-25 18:08:23,937::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:08:23,957::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00343174 s, 164 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:23,958::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:23,958::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:23,958::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:23,959::__init__::232::IOProcess::(_processLogs) (110573) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:23,962::__init__::232::IOProcess::(_processLogs) (110573) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:23,963::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:23,963::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:23,964::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:23,964::__init__::232::IOProcess::(_processLogs) (110574) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:23,965::__init__::232::IOProcess::(_processLogs) (110574) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:23,965::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:23,966::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:23,966::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:23,966::__init__::232::IOProcess::(_processLogs) (110575) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:23,967::__init__::232::IOProcess::(_processLogs) (110575) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:23,967::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:23,968::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:23,969::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:23,969::__init__::232::IOProcess::(_processLogs) (110576) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:23,969::__init__::232::IOProcess::(_processLogs) (110576) Queuing response >Dummy-148::DEBUG::2014-09-25 18:08:24,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:24,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013362 s, 76.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:08:26,536::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:26,544::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:26,545::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71775::DEBUG::2014-09-25 18:08:26,547::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:26,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152741 s, 67.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:27,595::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:27,595::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:27,595::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:27,596::__init__::232::IOProcess::(_processLogs) (110577) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:27,599::__init__::232::IOProcess::(_processLogs) (110577) Queuing response >Thread-81::DEBUG::2014-09-25 18:08:27,600::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:08:27,615::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00275173 s, 135 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:27,616::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:27,617::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:27,617::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:27,617::__init__::232::IOProcess::(_processLogs) (110578) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:27,619::__init__::232::IOProcess::(_processLogs) (110578) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:28,004::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:28,005::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:28,005::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:28,005::__init__::232::IOProcess::(_processLogs) (110579) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:08:28,006::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:28,006::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:28,006::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:28,006::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:28,007::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:28,007::__init__::232::IOProcess::(_processLogs) (110580) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:08:28,007::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:28,007::__init__::232::IOProcess::(_processLogs) (110581) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:08:28,008::__init__::232::IOProcess::(_processLogs) (110579) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:28,010::__init__::232::IOProcess::(_processLogs) (110581) Queuing response >Dummy-148::DEBUG::2014-09-25 18:08:28,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:28,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211604 s, 48.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:29,567::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:29,569::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71779::DEBUG::2014-09-25 18:08:29,571::task::595::Storage.TaskManager.Task::(_updateState) Task=`8e271300-e90a-4200-978e-9bbba0d44dcf`::moving from state init -> state preparing >Thread-71779::INFO::2014-09-25 18:08:29,571::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71779::INFO::2014-09-25 18:08:29,572::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00343174', 'lastCheck': '5.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '11.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00275173', 'lastCheck': '2.0', 'valid': True}} >Thread-71779::DEBUG::2014-09-25 18:08:29,572::task::1191::Storage.TaskManager.Task::(prepare) Task=`8e271300-e90a-4200-978e-9bbba0d44dcf`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00343174', 'lastCheck': '5.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '11.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00275173', 'lastCheck': '2.0', 'valid': True}} >Thread-71779::DEBUG::2014-09-25 18:08:29,572::task::595::Storage.TaskManager.Task::(_updateState) Task=`8e271300-e90a-4200-978e-9bbba0d44dcf`::moving from state preparing -> state finished >Thread-71779::DEBUG::2014-09-25 18:08:29,573::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71779::DEBUG::2014-09-25 18:08:29,573::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71779::DEBUG::2014-09-25 18:08:29,573::task::993::Storage.TaskManager.Task::(_decref) Task=`8e271300-e90a-4200-978e-9bbba0d44dcf`::ref 0 aborting False >Thread-71779::DEBUG::2014-09-25 18:08:29,575::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:29,593::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:29,594::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71780::DEBUG::2014-09-25 18:08:29,596::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:30,447::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71781::DEBUG::2014-09-25 18:08:30,448::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:08:30,448::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71781::DEBUG::2014-09-25 18:08:30,450::task::595::Storage.TaskManager.Task::(_updateState) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::moving from state init -> state preparing >Thread-71781::INFO::2014-09-25 18:08:30,451::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71781::ERROR::2014-09-25 18:08:30,451::task::866::Storage.TaskManager.Task::(_setError) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71781::DEBUG::2014-09-25 18:08:30,452::task::885::Storage.TaskManager.Task::(_run) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::Task._run: 359d9878-255c-4c4a-b6c8-362609120aeb (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71781::DEBUG::2014-09-25 18:08:30,452::task::1217::Storage.TaskManager.Task::(stop) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::stopping in state preparing (force False) >Thread-71781::DEBUG::2014-09-25 18:08:30,452::task::993::Storage.TaskManager.Task::(_decref) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::ref 1 aborting True >Thread-71781::INFO::2014-09-25 18:08:30,452::task::1171::Storage.TaskManager.Task::(prepare) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::aborting: Task is aborted: u'' - code 100 >Thread-71781::DEBUG::2014-09-25 18:08:30,453::task::1176::Storage.TaskManager.Task::(prepare) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::Prepare: aborted: >Thread-71781::DEBUG::2014-09-25 18:08:30,453::task::993::Storage.TaskManager.Task::(_decref) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::ref 0 aborting True >Thread-71781::DEBUG::2014-09-25 18:08:30,453::task::928::Storage.TaskManager.Task::(_doAbort) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::Task._doAbort: force False >Thread-71781::DEBUG::2014-09-25 18:08:30,453::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71781::DEBUG::2014-09-25 18:08:30,453::task::595::Storage.TaskManager.Task::(_updateState) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::moving from state preparing -> state aborting >Thread-71781::DEBUG::2014-09-25 18:08:30,454::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::_aborting: recover policy none >Thread-71781::DEBUG::2014-09-25 18:08:30,454::task::595::Storage.TaskManager.Task::(_updateState) Task=`359d9878-255c-4c4a-b6c8-362609120aeb`::moving from state aborting -> state failed >Thread-71781::DEBUG::2014-09-25 18:08:30,454::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71781::DEBUG::2014-09-25 18:08:30,454::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71781::ERROR::2014-09-25 18:08:30,454::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71781::DEBUG::2014-09-25 18:08:30,455::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:30,639::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:30,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209451 s, 48.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:32,637::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:32,638::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71782::DEBUG::2014-09-25 18:08:32,640::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:32,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:32,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0223373 s, 45.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:33,969::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:33,970::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:33,970::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:33,972::__init__::232::IOProcess::(_processLogs) (110582) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 18:08:33,972::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:08:33,973::__init__::232::IOProcess::(_processLogs) (110582) Queuing response >Thread-79::DEBUG::2014-09-25 18:08:33,982::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000397041 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:33,983::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:33,984::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:33,984::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:33,984::__init__::232::IOProcess::(_processLogs) (110583) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:33,985::__init__::232::IOProcess::(_processLogs) (110583) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:33,986::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:33,987::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:33,987::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:33,987::__init__::232::IOProcess::(_processLogs) (110584) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:33,988::__init__::232::IOProcess::(_processLogs) (110584) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:33,988::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:33,989::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:33,989::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:33,990::__init__::232::IOProcess::(_processLogs) (110585) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:33,990::__init__::232::IOProcess::(_processLogs) (110585) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:33,990::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:33,991::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:33,992::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:33,992::__init__::232::IOProcess::(_processLogs) (110586) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:33,992::__init__::232::IOProcess::(_processLogs) (110586) Queuing response >Dummy-148::DEBUG::2014-09-25 18:08:34,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:34,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134544 s, 76.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:35,653::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:35,654::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71783::DEBUG::2014-09-25 18:08:35,656::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:36,792::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:36,819::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183881 s, 55.7 MB/s\n'; <rc> = 0 >Thread-71759::WARNING::2014-09-25 18:08:37,438::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-29::DEBUG::2014-09-25 18:08:37,621::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:37,621::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:37,621::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:37,621::__init__::232::IOProcess::(_processLogs) (110587) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:37,624::__init__::232::IOProcess::(_processLogs) (110587) Queuing response >Thread-81::DEBUG::2014-09-25 18:08:37,625::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:08:37,642::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00632288 s, 58.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:37,643::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:37,644::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:37,644::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:37,644::__init__::232::IOProcess::(_processLogs) (110588) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:37,646::__init__::232::IOProcess::(_processLogs) (110588) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:38,673::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:38,674::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71784::DEBUG::2014-09-25 18:08:38,676::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:38,839::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:38,868::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182936 s, 56.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:40,506::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71785::DEBUG::2014-09-25 18:08:40,507::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:08:40,507::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71785::DEBUG::2014-09-25 18:08:40,510::task::595::Storage.TaskManager.Task::(_updateState) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::moving from state init -> state preparing >Thread-71785::INFO::2014-09-25 18:08:40,510::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71785::ERROR::2014-09-25 18:08:40,510::task::866::Storage.TaskManager.Task::(_setError) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71785::DEBUG::2014-09-25 18:08:40,511::task::885::Storage.TaskManager.Task::(_run) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::Task._run: e27387d9-28ad-4bd6-bc29-498c641d7bef (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71785::DEBUG::2014-09-25 18:08:40,511::task::1217::Storage.TaskManager.Task::(stop) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::stopping in state preparing (force False) >Thread-71785::DEBUG::2014-09-25 18:08:40,511::task::993::Storage.TaskManager.Task::(_decref) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::ref 1 aborting True >Thread-71785::INFO::2014-09-25 18:08:40,511::task::1171::Storage.TaskManager.Task::(prepare) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::aborting: Task is aborted: u'' - code 100 >Thread-71785::DEBUG::2014-09-25 18:08:40,512::task::1176::Storage.TaskManager.Task::(prepare) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::Prepare: aborted: >Thread-71785::DEBUG::2014-09-25 18:08:40,512::task::993::Storage.TaskManager.Task::(_decref) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::ref 0 aborting True >Thread-71785::DEBUG::2014-09-25 18:08:40,512::task::928::Storage.TaskManager.Task::(_doAbort) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::Task._doAbort: force False >Thread-71785::DEBUG::2014-09-25 18:08:40,512::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71785::DEBUG::2014-09-25 18:08:40,513::task::595::Storage.TaskManager.Task::(_updateState) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::moving from state preparing -> state aborting >Thread-71785::DEBUG::2014-09-25 18:08:40,513::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::_aborting: recover policy none >Thread-71785::DEBUG::2014-09-25 18:08:40,513::task::595::Storage.TaskManager.Task::(_updateState) Task=`e27387d9-28ad-4bd6-bc29-498c641d7bef`::moving from state aborting -> state failed >Thread-71785::DEBUG::2014-09-25 18:08:40,513::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71785::DEBUG::2014-09-25 18:08:40,514::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71785::ERROR::2014-09-25 18:08:40,514::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71785::DEBUG::2014-09-25 18:08:40,514::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:40,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:40,938::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0336656 s, 30.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:41,696::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:41,697::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71786::DEBUG::2014-09-25 18:08:41,699::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:43,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:43,078::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021677 s, 47.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:43,992::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:43,993::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:43,993::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:43,993::__init__::232::IOProcess::(_processLogs) (110589) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:43,997::__init__::232::IOProcess::(_processLogs) (110589) Queuing response >Thread-79::DEBUG::2014-09-25 18:08:43,998::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:08:44,012::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00210381 s, 267 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:44,013::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:44,013::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:44,013::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:44,014::__init__::232::IOProcess::(_processLogs) (110590) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:44,019::__init__::232::IOProcess::(_processLogs) (110590) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:44,020::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:44,020::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:44,021::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:44,022::__init__::232::IOProcess::(_processLogs) (110591) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:44,022::__init__::232::IOProcess::(_processLogs) (110591) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:44,023::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:44,024::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:44,025::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:44,025::__init__::232::IOProcess::(_processLogs) (110592) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:44,025::__init__::232::IOProcess::(_processLogs) (110592) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:44,026::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:44,027::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:44,028::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:44,028::__init__::232::IOProcess::(_processLogs) (110593) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:44,028::__init__::232::IOProcess::(_processLogs) (110593) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:44,713::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:44,714::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71787::DEBUG::2014-09-25 18:08:44,717::task::595::Storage.TaskManager.Task::(_updateState) Task=`60e9daca-a12c-495b-88a7-00d3aa8a7874`::moving from state init -> state preparing >Thread-71787::INFO::2014-09-25 18:08:44,717::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71787::INFO::2014-09-25 18:08:44,718::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00210381', 'lastCheck': '0.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '26.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00632288', 'lastCheck': '7.1', 'valid': True}} >Thread-71787::DEBUG::2014-09-25 18:08:44,718::task::1191::Storage.TaskManager.Task::(prepare) Task=`60e9daca-a12c-495b-88a7-00d3aa8a7874`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00210381', 'lastCheck': '0.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '26.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00632288', 'lastCheck': '7.1', 'valid': True}} >Thread-71787::DEBUG::2014-09-25 18:08:44,718::task::595::Storage.TaskManager.Task::(_updateState) Task=`60e9daca-a12c-495b-88a7-00d3aa8a7874`::moving from state preparing -> state finished >Thread-71787::DEBUG::2014-09-25 18:08:44,719::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71787::DEBUG::2014-09-25 18:08:44,719::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71787::DEBUG::2014-09-25 18:08:44,719::task::993::Storage.TaskManager.Task::(_decref) Task=`60e9daca-a12c-495b-88a7-00d3aa8a7874`::ref 0 aborting False >Thread-71787::DEBUG::2014-09-25 18:08:44,721::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:44,737::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:44,738::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71788::DEBUG::2014-09-25 18:08:44,740::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:45,098::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:45,120::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132647 s, 77.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:08:47,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:47,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129619 s, 79.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:47,648::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:47,648::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:47,649::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:47,650::__init__::232::IOProcess::(_processLogs) (110594) Got request for method 'statvfs' >Thread-81::DEBUG::2014-09-25 18:08:47,651::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:08:47,651::__init__::232::IOProcess::(_processLogs) (110594) Queuing response >Thread-81::DEBUG::2014-09-25 18:08:47,662::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00174873 s, 212 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:47,664::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:47,664::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:47,664::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:47,665::__init__::232::IOProcess::(_processLogs) (110595) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:47,666::__init__::232::IOProcess::(_processLogs) (110595) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:47,790::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:47,792::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71789::DEBUG::2014-09-25 18:08:47,793::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:49,181::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:49,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129934 s, 78.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:50,544::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71790::DEBUG::2014-09-25 18:08:50,546::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:08:50,546::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71790::DEBUG::2014-09-25 18:08:50,548::task::595::Storage.TaskManager.Task::(_updateState) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::moving from state init -> state preparing >Thread-71790::INFO::2014-09-25 18:08:50,548::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71790::ERROR::2014-09-25 18:08:50,549::task::866::Storage.TaskManager.Task::(_setError) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71790::DEBUG::2014-09-25 18:08:50,549::task::885::Storage.TaskManager.Task::(_run) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::Task._run: 05918829-8969-4b14-9014-d6787f0c32c3 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71790::DEBUG::2014-09-25 18:08:50,549::task::1217::Storage.TaskManager.Task::(stop) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::stopping in state preparing (force False) >Thread-71790::DEBUG::2014-09-25 18:08:50,550::task::993::Storage.TaskManager.Task::(_decref) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::ref 1 aborting True >Thread-71790::INFO::2014-09-25 18:08:50,550::task::1171::Storage.TaskManager.Task::(prepare) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::aborting: Task is aborted: u'' - code 100 >Thread-71790::DEBUG::2014-09-25 18:08:50,550::task::1176::Storage.TaskManager.Task::(prepare) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::Prepare: aborted: >Thread-71790::DEBUG::2014-09-25 18:08:50,550::task::993::Storage.TaskManager.Task::(_decref) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::ref 0 aborting True >Thread-71790::DEBUG::2014-09-25 18:08:50,551::task::928::Storage.TaskManager.Task::(_doAbort) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::Task._doAbort: force False >Thread-71790::DEBUG::2014-09-25 18:08:50,551::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71790::DEBUG::2014-09-25 18:08:50,551::task::595::Storage.TaskManager.Task::(_updateState) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::moving from state preparing -> state aborting >Thread-71790::DEBUG::2014-09-25 18:08:50,551::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::_aborting: recover policy none >Thread-71790::DEBUG::2014-09-25 18:08:50,552::task::595::Storage.TaskManager.Task::(_updateState) Task=`05918829-8969-4b14-9014-d6787f0c32c3`::moving from state aborting -> state failed >Thread-71790::DEBUG::2014-09-25 18:08:50,552::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71790::DEBUG::2014-09-25 18:08:50,552::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71790::ERROR::2014-09-25 18:08:50,552::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71790::DEBUG::2014-09-25 18:08:50,553::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:50,814::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:50,815::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71791::DEBUG::2014-09-25 18:08:50,816::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:51,223::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:51,248::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148936 s, 68.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:08:53,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:53,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156971 s, 65.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:53,977::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:53,978::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71792::DEBUG::2014-09-25 18:08:53,980::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-79::DEBUG::2014-09-25 18:08:54,028::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 18:08:54,032::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,032::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,032::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,032::__init__::232::IOProcess::(_processLogs) (110596) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:08:54,033::__init__::232::IOProcess::(_processLogs) (110596) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:54,033::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,034::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,034::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,034::__init__::232::IOProcess::(_processLogs) (110597) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:08:54,035::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,035::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,035::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,036::__init__::232::IOProcess::(_processLogs) (110598) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:08:54,036::__init__::232::IOProcess::(_processLogs) (110598) Queuing response >Thread-79::DEBUG::2014-09-25 18:08:54,040::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 18:08:54,041::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,041::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,041::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,042::__init__::232::IOProcess::(_processLogs) (110599) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 18:08:54,063::__init__::232::IOProcess::(_processLogs) (110599) Queuing response >Thread-79::DEBUG::2014-09-25 18:08:54,063::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:08:54,064::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,064::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,064::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,065::__init__::232::IOProcess::(_processLogs) (110600) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 18:08:54,065::__init__::232::IOProcess::(_processLogs) (110600) Queuing response >Thread-79::DEBUG::2014-09-25 18:08:54,065::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 18:08:54,067::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,068::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,068::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,068::__init__::232::IOProcess::(_processLogs) (110601) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:54,069::__init__::232::IOProcess::(_processLogs) (110601) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:54,069::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,069::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,070::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,070::__init__::232::IOProcess::(_processLogs) (110602) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:08:54,070::__init__::232::IOProcess::(_processLogs) (110602) Queuing response >Thread-79::DEBUG::2014-09-25 18:08:54,071::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 18:08:54,071::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 18:08:54,072::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 18:08:54,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,073::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,073::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,073::__init__::232::IOProcess::(_processLogs) (110603) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:54,074::__init__::232::IOProcess::(_processLogs) (110603) Queuing response >Thread-79::DEBUG::2014-09-25 18:08:54,075::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:08:54,083::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00028184 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:54,084::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,085::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,085::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,085::__init__::232::IOProcess::(_processLogs) (110604) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:54,086::__init__::232::IOProcess::(_processLogs) (110604) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:54,087::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,087::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,088::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,088::__init__::232::IOProcess::(_processLogs) (110605) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:54,088::__init__::232::IOProcess::(_processLogs) (110605) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:54,089::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,089::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,090::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,090::__init__::232::IOProcess::(_processLogs) (110606) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:54,090::__init__::232::IOProcess::(_processLogs) (110606) Queuing response >Thread-29::DEBUG::2014-09-25 18:08:54,091::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:54,091::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:54,092::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:54,092::__init__::232::IOProcess::(_processLogs) (110607) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:08:54,092::__init__::232::IOProcess::(_processLogs) (110607) Queuing response >Dummy-148::DEBUG::2014-09-25 18:08:55,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:55,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146879 s, 69.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:08:57,007::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:08:57,009::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71796::DEBUG::2014-09-25 18:08:57,012::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:08:57,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:57,401::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0236325 s, 43.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:57,668::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:57,668::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:57,669::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:57,669::__init__::232::IOProcess::(_processLogs) (110608) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:57,669::__init__::232::IOProcess::(_processLogs) (110608) Queuing response >Thread-81::DEBUG::2014-09-25 18:08:57,670::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:08:57,678::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000470735 s, 788 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:08:57,680::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:08:57,680::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:08:57,680::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:08:57,680::__init__::232::IOProcess::(_processLogs) (110609) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:08:57,681::__init__::232::IOProcess::(_processLogs) (110609) Queuing response >Dummy-148::DEBUG::2014-09-25 18:08:59,421::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:08:59,443::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0128454 s, 79.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:00,026::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:00,027::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71797::DEBUG::2014-09-25 18:09:00,030::task::595::Storage.TaskManager.Task::(_updateState) Task=`fa562f2a-4011-4646-880b-7aad8c1a7fd7`::moving from state init -> state preparing >Thread-71797::INFO::2014-09-25 18:09:00,030::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71797::INFO::2014-09-25 18:09:00,031::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00028184', 'lastCheck': '5.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '42.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000470735', 'lastCheck': '2.3', 'valid': True}} >Thread-71797::DEBUG::2014-09-25 18:09:00,031::task::1191::Storage.TaskManager.Task::(prepare) Task=`fa562f2a-4011-4646-880b-7aad8c1a7fd7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00028184', 'lastCheck': '5.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '42.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000470735', 'lastCheck': '2.3', 'valid': True}} >Thread-71797::DEBUG::2014-09-25 18:09:00,031::task::595::Storage.TaskManager.Task::(_updateState) Task=`fa562f2a-4011-4646-880b-7aad8c1a7fd7`::moving from state preparing -> state finished >Thread-71797::DEBUG::2014-09-25 18:09:00,031::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71797::DEBUG::2014-09-25 18:09:00,031::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71797::DEBUG::2014-09-25 18:09:00,032::task::993::Storage.TaskManager.Task::(_decref) Task=`fa562f2a-4011-4646-880b-7aad8c1a7fd7`::ref 0 aborting False >Thread-71797::DEBUG::2014-09-25 18:09:00,034::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:00,052::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:00,054::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71798::DEBUG::2014-09-25 18:09:00,055::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:00,588::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71799::DEBUG::2014-09-25 18:09:00,589::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:09:00,589::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71799::DEBUG::2014-09-25 18:09:00,591::task::595::Storage.TaskManager.Task::(_updateState) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::moving from state init -> state preparing >Thread-71799::INFO::2014-09-25 18:09:00,592::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71799::ERROR::2014-09-25 18:09:00,592::task::866::Storage.TaskManager.Task::(_setError) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71799::DEBUG::2014-09-25 18:09:00,593::task::885::Storage.TaskManager.Task::(_run) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::Task._run: dca19655-8a89-4f39-b4fa-beadbe9dca79 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71799::DEBUG::2014-09-25 18:09:00,593::task::1217::Storage.TaskManager.Task::(stop) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::stopping in state preparing (force False) >Thread-71799::DEBUG::2014-09-25 18:09:00,593::task::993::Storage.TaskManager.Task::(_decref) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::ref 1 aborting True >Thread-71799::INFO::2014-09-25 18:09:00,593::task::1171::Storage.TaskManager.Task::(prepare) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::aborting: Task is aborted: u'' - code 100 >Thread-71799::DEBUG::2014-09-25 18:09:00,593::task::1176::Storage.TaskManager.Task::(prepare) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::Prepare: aborted: >Thread-71799::DEBUG::2014-09-25 18:09:00,594::task::993::Storage.TaskManager.Task::(_decref) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::ref 0 aborting True >Thread-71799::DEBUG::2014-09-25 18:09:00,594::task::928::Storage.TaskManager.Task::(_doAbort) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::Task._doAbort: force False >Thread-71799::DEBUG::2014-09-25 18:09:00,594::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71799::DEBUG::2014-09-25 18:09:00,594::task::595::Storage.TaskManager.Task::(_updateState) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::moving from state preparing -> state aborting >Thread-71799::DEBUG::2014-09-25 18:09:00,595::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::_aborting: recover policy none >Thread-71799::DEBUG::2014-09-25 18:09:00,595::task::595::Storage.TaskManager.Task::(_updateState) Task=`dca19655-8a89-4f39-b4fa-beadbe9dca79`::moving from state aborting -> state failed >Thread-71799::DEBUG::2014-09-25 18:09:00,595::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71799::DEBUG::2014-09-25 18:09:00,595::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71799::ERROR::2014-09-25 18:09:00,595::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71799::DEBUG::2014-09-25 18:09:00,596::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:01,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:01,726::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.254461 s, 4.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:03,107::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:03,108::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71800::DEBUG::2014-09-25 18:09:03,110::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:03,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:03,787::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.02158 s, 47.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:04,094::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:04,094::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:04,095::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:04,095::__init__::232::IOProcess::(_processLogs) (110610) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:04,096::__init__::232::IOProcess::(_processLogs) (110610) Queuing response >Thread-79::DEBUG::2014-09-25 18:09:04,097::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:09:04,111::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00195446 s, 288 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:04,112::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:04,112::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:04,112::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:04,113::__init__::232::IOProcess::(_processLogs) (110611) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:04,128::__init__::232::IOProcess::(_processLogs) (110611) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:04,129::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:04,130::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:04,130::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:04,130::__init__::232::IOProcess::(_processLogs) (110612) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:04,131::__init__::232::IOProcess::(_processLogs) (110612) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:04,131::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:04,132::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:04,132::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:04,133::__init__::232::IOProcess::(_processLogs) (110613) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:04,133::__init__::232::IOProcess::(_processLogs) (110613) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:04,133::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:04,134::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:04,134::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:04,135::__init__::232::IOProcess::(_processLogs) (110614) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:04,135::__init__::232::IOProcess::(_processLogs) (110614) Queuing response >Dummy-148::DEBUG::2014-09-25 18:09:05,807::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:05,834::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157133 s, 65.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:06,131::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:06,132::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71801::DEBUG::2014-09-25 18:09:06,135::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:09:07,683::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:07,684::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:07,684::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:07,684::__init__::232::IOProcess::(_processLogs) (110615) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:07,707::__init__::232::IOProcess::(_processLogs) (110615) Queuing response >Thread-81::DEBUG::2014-09-25 18:09:07,709::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:09:07,740::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00537855 s, 69.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:07,741::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:07,741::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:07,742::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:07,742::__init__::232::IOProcess::(_processLogs) (110616) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:07,745::__init__::232::IOProcess::(_processLogs) (110616) Queuing response >Dummy-148::DEBUG::2014-09-25 18:09:07,854::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:07,885::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0208759 s, 49.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:09,159::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:09,161::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71802::DEBUG::2014-09-25 18:09:09,163::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:09,905::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:09,933::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191444 s, 53.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:10,623::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71803::DEBUG::2014-09-25 18:09:10,624::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:09:10,624::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71803::DEBUG::2014-09-25 18:09:10,627::task::595::Storage.TaskManager.Task::(_updateState) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::moving from state init -> state preparing >Thread-71803::INFO::2014-09-25 18:09:10,627::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71803::ERROR::2014-09-25 18:09:10,627::task::866::Storage.TaskManager.Task::(_setError) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71803::DEBUG::2014-09-25 18:09:10,628::task::885::Storage.TaskManager.Task::(_run) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::Task._run: 99862a58-4191-46e7-889a-efc799c8d5f6 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71803::DEBUG::2014-09-25 18:09:10,628::task::1217::Storage.TaskManager.Task::(stop) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::stopping in state preparing (force False) >Thread-71803::DEBUG::2014-09-25 18:09:10,628::task::993::Storage.TaskManager.Task::(_decref) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::ref 1 aborting True >Thread-71803::INFO::2014-09-25 18:09:10,629::task::1171::Storage.TaskManager.Task::(prepare) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::aborting: Task is aborted: u'' - code 100 >Thread-71803::DEBUG::2014-09-25 18:09:10,629::task::1176::Storage.TaskManager.Task::(prepare) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::Prepare: aborted: >Thread-71803::DEBUG::2014-09-25 18:09:10,629::task::993::Storage.TaskManager.Task::(_decref) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::ref 0 aborting True >Thread-71803::DEBUG::2014-09-25 18:09:10,629::task::928::Storage.TaskManager.Task::(_doAbort) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::Task._doAbort: force False >Thread-71803::DEBUG::2014-09-25 18:09:10,630::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71803::DEBUG::2014-09-25 18:09:10,630::task::595::Storage.TaskManager.Task::(_updateState) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::moving from state preparing -> state aborting >Thread-71803::DEBUG::2014-09-25 18:09:10,630::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::_aborting: recover policy none >Thread-71803::DEBUG::2014-09-25 18:09:10,630::task::595::Storage.TaskManager.Task::(_updateState) Task=`99862a58-4191-46e7-889a-efc799c8d5f6`::moving from state aborting -> state failed >Thread-71803::DEBUG::2014-09-25 18:09:10,630::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71803::DEBUG::2014-09-25 18:09:10,631::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71803::ERROR::2014-09-25 18:09:10,631::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71803::DEBUG::2014-09-25 18:09:10,631::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:11,952::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:11,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150287 s, 68.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:12,181::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:12,182::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71804::DEBUG::2014-09-25 18:09:12,185::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:14,000::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:14,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179601 s, 57.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:14,136::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:14,136::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:14,136::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:14,137::__init__::232::IOProcess::(_processLogs) (110617) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:14,141::__init__::232::IOProcess::(_processLogs) (110617) Queuing response >Thread-79::DEBUG::2014-09-25 18:09:14,142::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:09:14,157::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00193627 s, 290 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:14,159::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:14,159::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:14,159::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:14,159::__init__::232::IOProcess::(_processLogs) (110618) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:14,162::__init__::232::IOProcess::(_processLogs) (110618) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:14,163::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:14,164::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:14,164::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:14,165::__init__::232::IOProcess::(_processLogs) (110619) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:14,165::__init__::232::IOProcess::(_processLogs) (110619) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:14,165::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:14,166::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:14,167::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:14,167::__init__::232::IOProcess::(_processLogs) (110620) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:14,167::__init__::232::IOProcess::(_processLogs) (110620) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:14,168::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:14,168::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:14,169::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:14,169::__init__::232::IOProcess::(_processLogs) (110621) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:14,170::__init__::232::IOProcess::(_processLogs) (110621) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:15,206::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:15,208::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71805::DEBUG::2014-09-25 18:09:15,210::task::595::Storage.TaskManager.Task::(_updateState) Task=`ee249a72-ea4d-4047-aa3d-c91d78774e8e`::moving from state init -> state preparing >Thread-71805::INFO::2014-09-25 18:09:15,211::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71805::INFO::2014-09-25 18:09:15,211::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00193627', 'lastCheck': '1.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '57.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00537855', 'lastCheck': '7.5', 'valid': True}} >Thread-71805::DEBUG::2014-09-25 18:09:15,212::task::1191::Storage.TaskManager.Task::(prepare) Task=`ee249a72-ea4d-4047-aa3d-c91d78774e8e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00193627', 'lastCheck': '1.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '57.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00537855', 'lastCheck': '7.5', 'valid': True}} >Thread-71805::DEBUG::2014-09-25 18:09:15,212::task::595::Storage.TaskManager.Task::(_updateState) Task=`ee249a72-ea4d-4047-aa3d-c91d78774e8e`::moving from state preparing -> state finished >Thread-71805::DEBUG::2014-09-25 18:09:15,212::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71805::DEBUG::2014-09-25 18:09:15,212::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71805::DEBUG::2014-09-25 18:09:15,212::task::993::Storage.TaskManager.Task::(_decref) Task=`ee249a72-ea4d-4047-aa3d-c91d78774e8e`::ref 0 aborting False >Thread-71805::DEBUG::2014-09-25 18:09:15,215::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:15,234::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:15,235::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71806::DEBUG::2014-09-25 18:09:15,237::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:16,055::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:16,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0307232 s, 33.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:17,747::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:17,747::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:17,747::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:17,747::__init__::232::IOProcess::(_processLogs) (110622) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:17,754::__init__::232::IOProcess::(_processLogs) (110622) Queuing response >Thread-81::DEBUG::2014-09-25 18:09:17,755::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:09:17,769::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0036602 s, 101 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:17,771::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:17,771::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:17,771::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:17,772::__init__::232::IOProcess::(_processLogs) (110623) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:17,774::__init__::232::IOProcess::(_processLogs) (110623) Queuing response >Dummy-148::DEBUG::2014-09-25 18:09:18,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:18,303::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:18,305::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71807::DEBUG::2014-09-25 18:09:18,307::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:18,329::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0223847 s, 45.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:09:20,349::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:20,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132718 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:20,658::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:20,660::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71808::DEBUG::2014-09-25 18:09:20,660::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71808::DEBUG::2014-09-25 18:09:20,662::task::595::Storage.TaskManager.Task::(_updateState) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::moving from state init -> state preparing >Thread-71808::INFO::2014-09-25 18:09:20,664::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71808::ERROR::2014-09-25 18:09:20,664::task::866::Storage.TaskManager.Task::(_setError) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71808::DEBUG::2014-09-25 18:09:20,665::task::885::Storage.TaskManager.Task::(_run) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::Task._run: 6e648246-a0a0-4359-9f1d-9eb899012805 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71808::DEBUG::2014-09-25 18:09:20,665::task::1217::Storage.TaskManager.Task::(stop) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::stopping in state preparing (force False) >Thread-71808::DEBUG::2014-09-25 18:09:20,665::task::993::Storage.TaskManager.Task::(_decref) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::ref 1 aborting True >Thread-71808::INFO::2014-09-25 18:09:20,665::task::1171::Storage.TaskManager.Task::(prepare) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::aborting: Task is aborted: u'' - code 100 >Thread-71808::DEBUG::2014-09-25 18:09:20,665::task::1176::Storage.TaskManager.Task::(prepare) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::Prepare: aborted: >Thread-71808::DEBUG::2014-09-25 18:09:20,666::task::993::Storage.TaskManager.Task::(_decref) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::ref 0 aborting True >Thread-71808::DEBUG::2014-09-25 18:09:20,666::task::928::Storage.TaskManager.Task::(_doAbort) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::Task._doAbort: force False >Thread-71808::DEBUG::2014-09-25 18:09:20,666::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71808::DEBUG::2014-09-25 18:09:20,666::task::595::Storage.TaskManager.Task::(_updateState) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::moving from state preparing -> state aborting >Thread-71808::DEBUG::2014-09-25 18:09:20,667::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::_aborting: recover policy none >Thread-71808::DEBUG::2014-09-25 18:09:20,667::task::595::Storage.TaskManager.Task::(_updateState) Task=`6e648246-a0a0-4359-9f1d-9eb899012805`::moving from state aborting -> state failed >Thread-71808::DEBUG::2014-09-25 18:09:20,667::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71808::DEBUG::2014-09-25 18:09:20,667::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71808::ERROR::2014-09-25 18:09:20,667::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71808::DEBUG::2014-09-25 18:09:20,668::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:21,323::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:21,325::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71809::DEBUG::2014-09-25 18:09:21,327::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:22,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:22,420::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169442 s, 60.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:24,170::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:24,171::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:24,171::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:24,171::__init__::232::IOProcess::(_processLogs) (110624) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:24,176::__init__::232::IOProcess::(_processLogs) (110624) Queuing response >Thread-79::DEBUG::2014-09-25 18:09:24,177::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:09:24,201::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00871226 s, 64.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:24,202::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:24,203::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:24,203::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:24,203::__init__::232::IOProcess::(_processLogs) (110625) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:24,210::__init__::232::IOProcess::(_processLogs) (110625) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:24,211::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:24,212::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:24,212::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:24,213::__init__::232::IOProcess::(_processLogs) (110626) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:24,213::__init__::232::IOProcess::(_processLogs) (110626) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:24,214::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:24,214::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:24,214::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:24,215::__init__::232::IOProcess::(_processLogs) (110627) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:24,215::__init__::232::IOProcess::(_processLogs) (110627) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:24,216::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:24,216::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:24,216::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:24,217::__init__::232::IOProcess::(_processLogs) (110628) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:24,218::__init__::232::IOProcess::(_processLogs) (110628) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:24,341::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:24,343::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71810::DEBUG::2014-09-25 18:09:24,345::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:24,440::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:24,478::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218127 s, 46.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:09:26,498::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:26,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.218816 s, 4.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:27,360::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:27,362::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71811::DEBUG::2014-09-25 18:09:27,364::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:09:27,776::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:27,776::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:27,777::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:27,777::__init__::232::IOProcess::(_processLogs) (110629) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:27,781::__init__::232::IOProcess::(_processLogs) (110629) Queuing response >Thread-81::DEBUG::2014-09-25 18:09:27,782::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:09:27,799::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00412714 s, 89.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:27,800::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:27,800::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:27,801::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:27,801::__init__::232::IOProcess::(_processLogs) (110630) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:27,804::__init__::232::IOProcess::(_processLogs) (110630) Queuing response >Thread-71777::WARNING::2014-09-25 18:09:28,004::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:09:28,005::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:09:28,005::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Dummy-148::DEBUG::2014-09-25 18:09:28,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:28,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161836 s, 63.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:30,386::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:30,387::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71812::DEBUG::2014-09-25 18:09:30,390::task::595::Storage.TaskManager.Task::(_updateState) Task=`20a4cf07-0fcc-47dc-9265-d0df6db70672`::moving from state init -> state preparing >Thread-71812::INFO::2014-09-25 18:09:30,390::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71812::INFO::2014-09-25 18:09:30,391::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00871226', 'lastCheck': '6.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.4', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00412714', 'lastCheck': '2.6', 'valid': True}} >Thread-71812::DEBUG::2014-09-25 18:09:30,391::task::1191::Storage.TaskManager.Task::(prepare) Task=`20a4cf07-0fcc-47dc-9265-d0df6db70672`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00871226', 'lastCheck': '6.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '2.4', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00412714', 'lastCheck': '2.6', 'valid': True}} >Thread-71812::DEBUG::2014-09-25 18:09:30,391::task::595::Storage.TaskManager.Task::(_updateState) Task=`20a4cf07-0fcc-47dc-9265-d0df6db70672`::moving from state preparing -> state finished >Thread-71812::DEBUG::2014-09-25 18:09:30,391::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71812::DEBUG::2014-09-25 18:09:30,391::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71812::DEBUG::2014-09-25 18:09:30,392::task::993::Storage.TaskManager.Task::(_decref) Task=`20a4cf07-0fcc-47dc-9265-d0df6db70672`::ref 0 aborting False >Thread-71812::DEBUG::2014-09-25 18:09:30,394::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:30,415::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:30,416::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71813::DEBUG::2014-09-25 18:09:30,418::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:30,694::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71814::DEBUG::2014-09-25 18:09:30,695::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:09:30,695::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71814::DEBUG::2014-09-25 18:09:30,697::task::595::Storage.TaskManager.Task::(_updateState) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::moving from state init -> state preparing >Thread-71814::INFO::2014-09-25 18:09:30,697::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71814::ERROR::2014-09-25 18:09:30,698::task::866::Storage.TaskManager.Task::(_setError) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71814::DEBUG::2014-09-25 18:09:30,698::task::885::Storage.TaskManager.Task::(_run) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::Task._run: a6f3b330-c6f9-47ba-805a-9ceadc6f9df7 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71814::DEBUG::2014-09-25 18:09:30,698::task::1217::Storage.TaskManager.Task::(stop) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::stopping in state preparing (force False) >Thread-71814::DEBUG::2014-09-25 18:09:30,699::task::993::Storage.TaskManager.Task::(_decref) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::ref 1 aborting True >Thread-71814::INFO::2014-09-25 18:09:30,699::task::1171::Storage.TaskManager.Task::(prepare) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::aborting: Task is aborted: u'' - code 100 >Thread-71814::DEBUG::2014-09-25 18:09:30,699::task::1176::Storage.TaskManager.Task::(prepare) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::Prepare: aborted: >Thread-71814::DEBUG::2014-09-25 18:09:30,699::task::993::Storage.TaskManager.Task::(_decref) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::ref 0 aborting True >Thread-71814::DEBUG::2014-09-25 18:09:30,699::task::928::Storage.TaskManager.Task::(_doAbort) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::Task._doAbort: force False >Thread-71814::DEBUG::2014-09-25 18:09:30,700::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71814::DEBUG::2014-09-25 18:09:30,700::task::595::Storage.TaskManager.Task::(_updateState) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::moving from state preparing -> state aborting >Thread-71814::DEBUG::2014-09-25 18:09:30,700::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::_aborting: recover policy none >Thread-71814::DEBUG::2014-09-25 18:09:30,700::task::595::Storage.TaskManager.Task::(_updateState) Task=`a6f3b330-c6f9-47ba-805a-9ceadc6f9df7`::moving from state aborting -> state failed >Thread-71814::DEBUG::2014-09-25 18:09:30,701::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71814::DEBUG::2014-09-25 18:09:30,701::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71814::ERROR::2014-09-25 18:09:30,701::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71814::DEBUG::2014-09-25 18:09:30,701::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:30,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:30,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013518 s, 75.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:09:32,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:32,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017138 s, 59.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:33,470::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:33,471::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71815::DEBUG::2014-09-25 18:09:33,473::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:09:34,218::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:34,218::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:34,219::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:34,219::__init__::232::IOProcess::(_processLogs) (110631) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:34,219::__init__::232::IOProcess::(_processLogs) (110631) Queuing response >Thread-79::DEBUG::2014-09-25 18:09:34,221::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:09:34,244::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00642247 s, 87.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:34,245::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:34,246::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:34,246::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:34,246::__init__::232::IOProcess::(_processLogs) (110632) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:34,251::__init__::232::IOProcess::(_processLogs) (110632) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:34,253::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:34,253::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:34,254::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:34,254::__init__::232::IOProcess::(_processLogs) (110633) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:34,254::__init__::232::IOProcess::(_processLogs) (110633) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:34,255::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:34,255::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:34,256::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:34,256::__init__::232::IOProcess::(_processLogs) (110634) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:34,256::__init__::232::IOProcess::(_processLogs) (110634) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:34,257::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:34,257::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:34,258::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:34,259::__init__::232::IOProcess::(_processLogs) (110635) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:34,259::__init__::232::IOProcess::(_processLogs) (110635) Queuing response >Dummy-148::DEBUG::2014-09-25 18:09:34,890::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:34,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191994 s, 53.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:36,488::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:36,489::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71816::DEBUG::2014-09-25 18:09:36,491::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:36,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:36,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212756 s, 48.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:37,806::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:37,807::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:37,807::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:37,807::__init__::232::IOProcess::(_processLogs) (110636) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:37,808::__init__::232::IOProcess::(_processLogs) (110636) Queuing response >Thread-81::DEBUG::2014-09-25 18:09:37,808::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:09:37,824::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00301927 s, 123 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:37,826::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:37,826::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:37,826::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:37,826::__init__::232::IOProcess::(_processLogs) (110637) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:37,833::__init__::232::IOProcess::(_processLogs) (110637) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:38,009::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:38,010::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:38,010::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:38,011::__init__::232::IOProcess::(_processLogs) (110638) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:09:38,012::__init__::232::IOProcess::(_processLogs) (110638) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:38,013::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:38,013::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:38,013::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:38,013::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:38,014::__init__::232::IOProcess::(_processLogs) (110639) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:09:38,014::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:38,014::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:38,014::__init__::232::IOProcess::(_processLogs) (110640) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:09:38,016::__init__::232::IOProcess::(_processLogs) (110640) Queuing response >Dummy-148::DEBUG::2014-09-25 18:09:38,991::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:39,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021219 s, 48.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:39,512::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:39,513::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71820::DEBUG::2014-09-25 18:09:39,515::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:40,729::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:40,730::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71821::DEBUG::2014-09-25 18:09:40,730::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71821::DEBUG::2014-09-25 18:09:40,732::task::595::Storage.TaskManager.Task::(_updateState) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::moving from state init -> state preparing >Thread-71821::INFO::2014-09-25 18:09:40,732::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71821::ERROR::2014-09-25 18:09:40,733::task::866::Storage.TaskManager.Task::(_setError) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71821::DEBUG::2014-09-25 18:09:40,733::task::885::Storage.TaskManager.Task::(_run) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::Task._run: 763a8e2c-0262-4f96-a3fb-74526afa68eb (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71821::DEBUG::2014-09-25 18:09:40,734::task::1217::Storage.TaskManager.Task::(stop) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::stopping in state preparing (force False) >Thread-71821::DEBUG::2014-09-25 18:09:40,734::task::993::Storage.TaskManager.Task::(_decref) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::ref 1 aborting True >Thread-71821::INFO::2014-09-25 18:09:40,734::task::1171::Storage.TaskManager.Task::(prepare) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::aborting: Task is aborted: u'' - code 100 >Thread-71821::DEBUG::2014-09-25 18:09:40,734::task::1176::Storage.TaskManager.Task::(prepare) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::Prepare: aborted: >Thread-71821::DEBUG::2014-09-25 18:09:40,734::task::993::Storage.TaskManager.Task::(_decref) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::ref 0 aborting True >Thread-71821::DEBUG::2014-09-25 18:09:40,735::task::928::Storage.TaskManager.Task::(_doAbort) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::Task._doAbort: force False >Thread-71821::DEBUG::2014-09-25 18:09:40,735::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71821::DEBUG::2014-09-25 18:09:40,735::task::595::Storage.TaskManager.Task::(_updateState) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::moving from state preparing -> state aborting >Thread-71821::DEBUG::2014-09-25 18:09:40,735::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::_aborting: recover policy none >Thread-71821::DEBUG::2014-09-25 18:09:40,736::task::595::Storage.TaskManager.Task::(_updateState) Task=`763a8e2c-0262-4f96-a3fb-74526afa68eb`::moving from state aborting -> state failed >Thread-71821::DEBUG::2014-09-25 18:09:40,736::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71821::DEBUG::2014-09-25 18:09:40,736::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71821::ERROR::2014-09-25 18:09:40,736::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71821::DEBUG::2014-09-25 18:09:40,737::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:41,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:41,075::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014605 s, 70.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:42,542::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:42,544::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71822::DEBUG::2014-09-25 18:09:42,546::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:43,095::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:43,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0227263 s, 45.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:44,259::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:44,260::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:44,260::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:44,260::__init__::232::IOProcess::(_processLogs) (110641) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:44,261::__init__::232::IOProcess::(_processLogs) (110641) Queuing response >Thread-79::DEBUG::2014-09-25 18:09:44,262::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:09:44,270::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000364314 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:44,272::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:44,272::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:44,273::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:44,274::__init__::232::IOProcess::(_processLogs) (110642) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:44,274::__init__::232::IOProcess::(_processLogs) (110642) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:44,274::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:44,275::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:44,276::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:44,276::__init__::232::IOProcess::(_processLogs) (110643) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:44,276::__init__::232::IOProcess::(_processLogs) (110643) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:44,277::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:44,277::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:44,278::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:44,278::__init__::232::IOProcess::(_processLogs) (110644) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:44,278::__init__::232::IOProcess::(_processLogs) (110644) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:44,279::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:44,279::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:44,280::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:44,281::__init__::232::IOProcess::(_processLogs) (110645) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:44,281::__init__::232::IOProcess::(_processLogs) (110645) Queuing response >Dummy-148::DEBUG::2014-09-25 18:09:45,152::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:45,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178865 s, 57.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:45,562::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:45,563::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71823::DEBUG::2014-09-25 18:09:45,567::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d5b4574-b2bb-4267-a4e2-f2771dbfde64`::moving from state init -> state preparing >Thread-71823::INFO::2014-09-25 18:09:45,567::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71823::INFO::2014-09-25 18:09:45,567::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000364314', 'lastCheck': '1.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '17.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00301927', 'lastCheck': '7.7', 'valid': True}} >Thread-71823::DEBUG::2014-09-25 18:09:45,568::task::1191::Storage.TaskManager.Task::(prepare) Task=`6d5b4574-b2bb-4267-a4e2-f2771dbfde64`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000364314', 'lastCheck': '1.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '17.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00301927', 'lastCheck': '7.7', 'valid': True}} >Thread-71823::DEBUG::2014-09-25 18:09:45,568::task::595::Storage.TaskManager.Task::(_updateState) Task=`6d5b4574-b2bb-4267-a4e2-f2771dbfde64`::moving from state preparing -> state finished >Thread-71823::DEBUG::2014-09-25 18:09:45,568::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71823::DEBUG::2014-09-25 18:09:45,568::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71823::DEBUG::2014-09-25 18:09:45,569::task::993::Storage.TaskManager.Task::(_decref) Task=`6d5b4574-b2bb-4267-a4e2-f2771dbfde64`::ref 0 aborting False >Thread-71823::DEBUG::2014-09-25 18:09:45,571::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:45,591::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:45,593::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71824::DEBUG::2014-09-25 18:09:45,594::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:47,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:47,225::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131499 s, 77.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:47,836::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:47,836::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:47,836::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:47,837::__init__::232::IOProcess::(_processLogs) (110646) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:47,843::__init__::232::IOProcess::(_processLogs) (110646) Queuing response >Thread-81::DEBUG::2014-09-25 18:09:47,843::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:09:47,856::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00346015 s, 107 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:47,858::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:47,858::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:47,858::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:47,858::__init__::232::IOProcess::(_processLogs) (110647) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:47,865::__init__::232::IOProcess::(_processLogs) (110647) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:48,646::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:48,647::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71825::DEBUG::2014-09-25 18:09:48,649::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:49,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:49,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167093 s, 61.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:50,764::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71826::DEBUG::2014-09-25 18:09:50,765::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:09:50,765::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71826::DEBUG::2014-09-25 18:09:50,768::task::595::Storage.TaskManager.Task::(_updateState) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::moving from state init -> state preparing >Thread-71826::INFO::2014-09-25 18:09:50,769::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71826::ERROR::2014-09-25 18:09:50,769::task::866::Storage.TaskManager.Task::(_setError) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71826::DEBUG::2014-09-25 18:09:50,770::task::885::Storage.TaskManager.Task::(_run) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::Task._run: 94149841-5ca3-4204-8cd6-68d9c99fa5f0 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71826::DEBUG::2014-09-25 18:09:50,770::task::1217::Storage.TaskManager.Task::(stop) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::stopping in state preparing (force False) >Thread-71826::DEBUG::2014-09-25 18:09:50,770::task::993::Storage.TaskManager.Task::(_decref) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::ref 1 aborting True >Thread-71826::INFO::2014-09-25 18:09:50,770::task::1171::Storage.TaskManager.Task::(prepare) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::aborting: Task is aborted: u'' - code 100 >Thread-71826::DEBUG::2014-09-25 18:09:50,771::task::1176::Storage.TaskManager.Task::(prepare) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::Prepare: aborted: >Thread-71826::DEBUG::2014-09-25 18:09:50,771::task::993::Storage.TaskManager.Task::(_decref) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::ref 0 aborting True >Thread-71826::DEBUG::2014-09-25 18:09:50,771::task::928::Storage.TaskManager.Task::(_doAbort) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::Task._doAbort: force False >Thread-71826::DEBUG::2014-09-25 18:09:50,771::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71826::DEBUG::2014-09-25 18:09:50,772::task::595::Storage.TaskManager.Task::(_updateState) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::moving from state preparing -> state aborting >Thread-71826::DEBUG::2014-09-25 18:09:50,772::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::_aborting: recover policy none >Thread-71826::DEBUG::2014-09-25 18:09:50,772::task::595::Storage.TaskManager.Task::(_updateState) Task=`94149841-5ca3-4204-8cd6-68d9c99fa5f0`::moving from state aborting -> state failed >Thread-71826::DEBUG::2014-09-25 18:09:50,772::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71826::DEBUG::2014-09-25 18:09:50,772::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71826::ERROR::2014-09-25 18:09:50,773::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71826::DEBUG::2014-09-25 18:09:50,773::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:51,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:51,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144817 s, 70.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:51,670::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:51,671::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71827::DEBUG::2014-09-25 18:09:51,674::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:53,341::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:53,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138692 s, 73.8 MB/s\n'; <rc> = 0 >Thread-71794::WARNING::2014-09-25 18:09:54,031::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-29::DEBUG::2014-09-25 18:09:54,281::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:54,281::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:54,282::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:54,282::__init__::232::IOProcess::(_processLogs) (110648) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:54,282::__init__::232::IOProcess::(_processLogs) (110648) Queuing response >Thread-79::DEBUG::2014-09-25 18:09:54,283::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:09:54,297::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00444195 s, 127 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:54,298::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:54,298::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:54,299::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:54,299::__init__::232::IOProcess::(_processLogs) (110649) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:54,300::__init__::232::IOProcess::(_processLogs) (110649) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:54,302::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:54,302::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:54,302::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:54,302::__init__::232::IOProcess::(_processLogs) (110650) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:54,308::__init__::232::IOProcess::(_processLogs) (110650) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:54,308::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:54,309::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:54,309::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:54,309::__init__::232::IOProcess::(_processLogs) (110651) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:54,312::__init__::232::IOProcess::(_processLogs) (110651) Queuing response >Thread-29::DEBUG::2014-09-25 18:09:54,313::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:54,313::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:54,313::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:54,314::__init__::232::IOProcess::(_processLogs) (110652) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:09:54,316::__init__::232::IOProcess::(_processLogs) (110652) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:54,688::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:54,689::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71828::DEBUG::2014-09-25 18:09:54,691::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:09:55,387::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:55,416::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015995 s, 64.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:09:57,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:57,462::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160732 s, 63.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:09:57,701::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:09:57,702::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71829::DEBUG::2014-09-25 18:09:57,705::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:09:57,868::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:57,868::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:57,868::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:57,868::__init__::232::IOProcess::(_processLogs) (110653) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:57,916::__init__::232::IOProcess::(_processLogs) (110653) Queuing response >Thread-81::DEBUG::2014-09-25 18:09:57,918::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:09:57,934::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00300148 s, 124 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:09:57,935::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:09:57,935::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:09:57,936::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:09:57,936::__init__::232::IOProcess::(_processLogs) (110654) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:09:57,942::__init__::232::IOProcess::(_processLogs) (110654) Queuing response >Dummy-148::DEBUG::2014-09-25 18:09:59,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:09:59,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213722 s, 47.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:00,720::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:00,722::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71830::DEBUG::2014-09-25 18:10:00,724::task::595::Storage.TaskManager.Task::(_updateState) Task=`65c6e3b7-710c-4c20-8a2c-41ce56ce4bef`::moving from state init -> state preparing >Thread-71830::INFO::2014-09-25 18:10:00,725::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71830::INFO::2014-09-25 18:10:00,725::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00444195', 'lastCheck': '6.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '32.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00300148', 'lastCheck': '2.8', 'valid': True}} >Thread-71830::DEBUG::2014-09-25 18:10:00,726::task::1191::Storage.TaskManager.Task::(prepare) Task=`65c6e3b7-710c-4c20-8a2c-41ce56ce4bef`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00444195', 'lastCheck': '6.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '32.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00300148', 'lastCheck': '2.8', 'valid': True}} >Thread-71830::DEBUG::2014-09-25 18:10:00,726::task::595::Storage.TaskManager.Task::(_updateState) Task=`65c6e3b7-710c-4c20-8a2c-41ce56ce4bef`::moving from state preparing -> state finished >Thread-71830::DEBUG::2014-09-25 18:10:00,726::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71830::DEBUG::2014-09-25 18:10:00,726::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71830::DEBUG::2014-09-25 18:10:00,726::task::993::Storage.TaskManager.Task::(_decref) Task=`65c6e3b7-710c-4c20-8a2c-41ce56ce4bef`::ref 0 aborting False >Thread-71830::DEBUG::2014-09-25 18:10:00,729::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:00,745::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:00,746::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71831::DEBUG::2014-09-25 18:10:00,747::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:00,806::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71832::DEBUG::2014-09-25 18:10:00,807::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:10:00,807::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71832::DEBUG::2014-09-25 18:10:00,809::task::595::Storage.TaskManager.Task::(_updateState) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::moving from state init -> state preparing >Thread-71832::INFO::2014-09-25 18:10:00,809::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71832::ERROR::2014-09-25 18:10:00,810::task::866::Storage.TaskManager.Task::(_setError) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71832::DEBUG::2014-09-25 18:10:00,810::task::885::Storage.TaskManager.Task::(_run) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::Task._run: 5dbc3eef-1e57-4f23-8fa6-38a47904271e (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71832::DEBUG::2014-09-25 18:10:00,811::task::1217::Storage.TaskManager.Task::(stop) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::stopping in state preparing (force False) >Thread-71832::DEBUG::2014-09-25 18:10:00,811::task::993::Storage.TaskManager.Task::(_decref) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::ref 1 aborting True >Thread-71832::INFO::2014-09-25 18:10:00,811::task::1171::Storage.TaskManager.Task::(prepare) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::aborting: Task is aborted: u'' - code 100 >Thread-71832::DEBUG::2014-09-25 18:10:00,811::task::1176::Storage.TaskManager.Task::(prepare) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::Prepare: aborted: >Thread-71832::DEBUG::2014-09-25 18:10:00,812::task::993::Storage.TaskManager.Task::(_decref) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::ref 0 aborting True >Thread-71832::DEBUG::2014-09-25 18:10:00,812::task::928::Storage.TaskManager.Task::(_doAbort) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::Task._doAbort: force False >Thread-71832::DEBUG::2014-09-25 18:10:00,812::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71832::DEBUG::2014-09-25 18:10:00,812::task::595::Storage.TaskManager.Task::(_updateState) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::moving from state preparing -> state aborting >Thread-71832::DEBUG::2014-09-25 18:10:00,812::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::_aborting: recover policy none >Thread-71832::DEBUG::2014-09-25 18:10:00,813::task::595::Storage.TaskManager.Task::(_updateState) Task=`5dbc3eef-1e57-4f23-8fa6-38a47904271e`::moving from state aborting -> state failed >Thread-71832::DEBUG::2014-09-25 18:10:00,813::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71832::DEBUG::2014-09-25 18:10:00,813::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71832::ERROR::2014-09-25 18:10:00,813::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71832::DEBUG::2014-09-25 18:10:00,814::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:01,535::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:01,564::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174981 s, 58.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:10:03,584::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:03,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211783 s, 48.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:03,802::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:03,804::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71833::DEBUG::2014-09-25 18:10:03,806::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:10:04,318::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:04,318::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:04,318::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:04,319::__init__::232::IOProcess::(_processLogs) (110655) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:04,319::__init__::232::IOProcess::(_processLogs) (110655) Queuing response >Thread-79::DEBUG::2014-09-25 18:10:04,320::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:10:04,341::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00333631 s, 168 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:04,342::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:04,342::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:04,343::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:04,343::__init__::232::IOProcess::(_processLogs) (110656) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:04,346::__init__::232::IOProcess::(_processLogs) (110656) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:04,348::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:04,348::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:04,349::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:04,349::__init__::232::IOProcess::(_processLogs) (110657) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:04,349::__init__::232::IOProcess::(_processLogs) (110657) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:04,350::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:04,350::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:04,351::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:04,351::__init__::232::IOProcess::(_processLogs) (110658) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:04,351::__init__::232::IOProcess::(_processLogs) (110658) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:04,352::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:04,352::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:04,353::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:04,353::__init__::232::IOProcess::(_processLogs) (110659) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:04,354::__init__::232::IOProcess::(_processLogs) (110659) Queuing response >Dummy-148::DEBUG::2014-09-25 18:10:05,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:05,665::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174514 s, 58.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:06,821::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:06,822::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71834::DEBUG::2014-09-25 18:10:06,824::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:07,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:07,713::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189903 s, 53.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:07,944::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:07,945::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:07,945::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:07,945::__init__::232::IOProcess::(_processLogs) (110660) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:07,953::__init__::232::IOProcess::(_processLogs) (110660) Queuing response >Thread-81::DEBUG::2014-09-25 18:10:07,955::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:10:07,982::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00136607 s, 272 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:07,983::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:07,984::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:07,984::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:07,984::__init__::232::IOProcess::(_processLogs) (110661) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:07,985::__init__::232::IOProcess::(_processLogs) (110661) Queuing response >Dummy-148::DEBUG::2014-09-25 18:10:09,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:09,759::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015364 s, 66.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:09,838::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:09,840::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71835::DEBUG::2014-09-25 18:10:09,842::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:10,833::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:10,834::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71836::DEBUG::2014-09-25 18:10:10,835::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71836::DEBUG::2014-09-25 18:10:10,837::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::moving from state init -> state preparing >Thread-71836::INFO::2014-09-25 18:10:10,837::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71836::ERROR::2014-09-25 18:10:10,837::task::866::Storage.TaskManager.Task::(_setError) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71836::DEBUG::2014-09-25 18:10:10,838::task::885::Storage.TaskManager.Task::(_run) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::Task._run: 2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71836::DEBUG::2014-09-25 18:10:10,838::task::1217::Storage.TaskManager.Task::(stop) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::stopping in state preparing (force False) >Thread-71836::DEBUG::2014-09-25 18:10:10,839::task::993::Storage.TaskManager.Task::(_decref) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::ref 1 aborting True >Thread-71836::INFO::2014-09-25 18:10:10,839::task::1171::Storage.TaskManager.Task::(prepare) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::aborting: Task is aborted: u'' - code 100 >Thread-71836::DEBUG::2014-09-25 18:10:10,839::task::1176::Storage.TaskManager.Task::(prepare) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::Prepare: aborted: >Thread-71836::DEBUG::2014-09-25 18:10:10,839::task::993::Storage.TaskManager.Task::(_decref) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::ref 0 aborting True >Thread-71836::DEBUG::2014-09-25 18:10:10,840::task::928::Storage.TaskManager.Task::(_doAbort) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::Task._doAbort: force False >Thread-71836::DEBUG::2014-09-25 18:10:10,840::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71836::DEBUG::2014-09-25 18:10:10,840::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::moving from state preparing -> state aborting >Thread-71836::DEBUG::2014-09-25 18:10:10,840::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::_aborting: recover policy none >Thread-71836::DEBUG::2014-09-25 18:10:10,840::task::595::Storage.TaskManager.Task::(_updateState) Task=`2d4f9dc5-23d4-4bcb-b6bd-a409ae0f0d25`::moving from state aborting -> state failed >Thread-71836::DEBUG::2014-09-25 18:10:10,841::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71836::DEBUG::2014-09-25 18:10:10,841::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71836::ERROR::2014-09-25 18:10:10,841::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71836::DEBUG::2014-09-25 18:10:10,842::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:11,793::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:11,817::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013479 s, 76.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:12,853::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:12,854::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71837::DEBUG::2014-09-25 18:10:12,856::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:13,837::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:13,864::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157997 s, 64.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:14,354::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:14,354::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:14,355::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:14,355::__init__::232::IOProcess::(_processLogs) (110662) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:14,360::__init__::232::IOProcess::(_processLogs) (110662) Queuing response >Thread-79::DEBUG::2014-09-25 18:10:14,361::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:10:14,375::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000608772 s, 923 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:14,377::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:14,377::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:14,377::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:14,377::__init__::232::IOProcess::(_processLogs) (110663) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:14,381::__init__::232::IOProcess::(_processLogs) (110663) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:14,382::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:14,382::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:14,383::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:14,383::__init__::232::IOProcess::(_processLogs) (110664) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:14,383::__init__::232::IOProcess::(_processLogs) (110664) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:14,384::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:14,385::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:14,385::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:14,385::__init__::232::IOProcess::(_processLogs) (110665) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:14,385::__init__::232::IOProcess::(_processLogs) (110665) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:14,388::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:14,390::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:14,390::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:14,390::__init__::232::IOProcess::(_processLogs) (110666) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:14,390::__init__::232::IOProcess::(_processLogs) (110666) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:15,865::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:15,867::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71838::DEBUG::2014-09-25 18:10:15,870::task::595::Storage.TaskManager.Task::(_updateState) Task=`c145f194-f263-43a6-8119-8b5aae24ba4a`::moving from state init -> state preparing >Thread-71838::INFO::2014-09-25 18:10:15,870::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71838::INFO::2014-09-25 18:10:15,870::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000608772', 'lastCheck': '1.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '47.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00136607', 'lastCheck': '7.9', 'valid': True}} >Thread-71838::DEBUG::2014-09-25 18:10:15,871::task::1191::Storage.TaskManager.Task::(prepare) Task=`c145f194-f263-43a6-8119-8b5aae24ba4a`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000608772', 'lastCheck': '1.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '47.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00136607', 'lastCheck': '7.9', 'valid': True}} >Thread-71838::DEBUG::2014-09-25 18:10:15,871::task::595::Storage.TaskManager.Task::(_updateState) Task=`c145f194-f263-43a6-8119-8b5aae24ba4a`::moving from state preparing -> state finished >Thread-71838::DEBUG::2014-09-25 18:10:15,871::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71838::DEBUG::2014-09-25 18:10:15,871::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71838::DEBUG::2014-09-25 18:10:15,872::task::993::Storage.TaskManager.Task::(_decref) Task=`c145f194-f263-43a6-8119-8b5aae24ba4a`::ref 0 aborting False >Thread-71838::DEBUG::2014-09-25 18:10:15,874::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:15,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:15,892::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:15,894::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71839::DEBUG::2014-09-25 18:10:15,896::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:15,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0195213 s, 52.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:10:17,935::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:17,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191966 s, 53.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:17,987::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:17,987::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:17,987::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:17,988::__init__::232::IOProcess::(_processLogs) (110667) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:18,001::__init__::232::IOProcess::(_processLogs) (110667) Queuing response >Thread-81::DEBUG::2014-09-25 18:10:18,002::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:10:18,021::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00514197 s, 72.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:18,022::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:18,022::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:18,023::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:18,023::__init__::232::IOProcess::(_processLogs) (110668) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:18,027::__init__::232::IOProcess::(_processLogs) (110668) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:18,951::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:18,953::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71840::DEBUG::2014-09-25 18:10:18,954::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:19,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:20,015::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182658 s, 56.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:20,872::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:20,873::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71841::DEBUG::2014-09-25 18:10:20,874::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71841::DEBUG::2014-09-25 18:10:20,876::task::595::Storage.TaskManager.Task::(_updateState) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::moving from state init -> state preparing >Thread-71841::INFO::2014-09-25 18:10:20,876::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71841::ERROR::2014-09-25 18:10:20,876::task::866::Storage.TaskManager.Task::(_setError) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71841::DEBUG::2014-09-25 18:10:20,877::task::885::Storage.TaskManager.Task::(_run) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::Task._run: ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71841::DEBUG::2014-09-25 18:10:20,877::task::1217::Storage.TaskManager.Task::(stop) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::stopping in state preparing (force False) >Thread-71841::DEBUG::2014-09-25 18:10:20,877::task::993::Storage.TaskManager.Task::(_decref) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::ref 1 aborting True >Thread-71841::INFO::2014-09-25 18:10:20,878::task::1171::Storage.TaskManager.Task::(prepare) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::aborting: Task is aborted: u'' - code 100 >Thread-71841::DEBUG::2014-09-25 18:10:20,878::task::1176::Storage.TaskManager.Task::(prepare) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::Prepare: aborted: >Thread-71841::DEBUG::2014-09-25 18:10:20,878::task::993::Storage.TaskManager.Task::(_decref) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::ref 0 aborting True >Thread-71841::DEBUG::2014-09-25 18:10:20,878::task::928::Storage.TaskManager.Task::(_doAbort) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::Task._doAbort: force False >Thread-71841::DEBUG::2014-09-25 18:10:20,879::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71841::DEBUG::2014-09-25 18:10:20,879::task::595::Storage.TaskManager.Task::(_updateState) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::moving from state preparing -> state aborting >Thread-71841::DEBUG::2014-09-25 18:10:20,879::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::_aborting: recover policy none >Thread-71841::DEBUG::2014-09-25 18:10:20,879::task::595::Storage.TaskManager.Task::(_updateState) Task=`ae8dddc7-8b4a-45ef-b84e-24dc0d9e3431`::moving from state aborting -> state failed >Thread-71841::DEBUG::2014-09-25 18:10:20,879::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71841::DEBUG::2014-09-25 18:10:20,880::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71841::ERROR::2014-09-25 18:10:20,880::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71841::DEBUG::2014-09-25 18:10:20,880::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:21,968::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:21,969::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71842::DEBUG::2014-09-25 18:10:21,971::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:22,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:22,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018135 s, 56.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:10:24,085::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:24,107::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133449 s, 76.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:24,390::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:24,391::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:24,391::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:24,391::__init__::232::IOProcess::(_processLogs) (110669) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:24,404::__init__::232::IOProcess::(_processLogs) (110669) Queuing response >Thread-79::DEBUG::2014-09-25 18:10:24,404::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:10:24,420::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00277987 s, 202 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:24,421::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:24,421::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:24,421::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:24,422::__init__::232::IOProcess::(_processLogs) (110670) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:24,424::__init__::232::IOProcess::(_processLogs) (110670) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:24,426::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:24,426::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:24,427::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:24,427::__init__::232::IOProcess::(_processLogs) (110671) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:24,427::__init__::232::IOProcess::(_processLogs) (110671) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:24,428::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:24,428::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:24,429::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:24,429::__init__::232::IOProcess::(_processLogs) (110672) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:24,429::__init__::232::IOProcess::(_processLogs) (110672) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:24,430::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:24,430::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:24,431::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:24,432::__init__::232::IOProcess::(_processLogs) (110673) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:24,432::__init__::232::IOProcess::(_processLogs) (110673) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:24,991::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:24,993::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71843::DEBUG::2014-09-25 18:10:24,995::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:26,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:26,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155717 s, 65.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:28,008::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:28,010::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71844::DEBUG::2014-09-25 18:10:28,012::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:10:28,029::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:28,029::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:28,029::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:28,029::__init__::232::IOProcess::(_processLogs) (110674) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:28,033::__init__::232::IOProcess::(_processLogs) (110674) Queuing response >Thread-81::DEBUG::2014-09-25 18:10:28,033::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:10:28,051::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0039765 s, 93.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:28,052::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:28,052::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:28,052::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:28,053::__init__::232::IOProcess::(_processLogs) (110675) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:28,055::__init__::232::IOProcess::(_processLogs) (110675) Queuing response >Dummy-148::DEBUG::2014-09-25 18:10:28,174::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:28,208::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0192893 s, 53.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:10:30,228::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:30,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132697 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:30,908::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:30,910::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71845::DEBUG::2014-09-25 18:10:30,910::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71845::DEBUG::2014-09-25 18:10:30,912::task::595::Storage.TaskManager.Task::(_updateState) Task=`30ece177-c342-47f7-82fd-385b1758527c`::moving from state init -> state preparing >Thread-71845::INFO::2014-09-25 18:10:30,912::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71845::ERROR::2014-09-25 18:10:30,913::task::866::Storage.TaskManager.Task::(_setError) Task=`30ece177-c342-47f7-82fd-385b1758527c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71845::DEBUG::2014-09-25 18:10:30,913::task::885::Storage.TaskManager.Task::(_run) Task=`30ece177-c342-47f7-82fd-385b1758527c`::Task._run: 30ece177-c342-47f7-82fd-385b1758527c (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71845::DEBUG::2014-09-25 18:10:30,913::task::1217::Storage.TaskManager.Task::(stop) Task=`30ece177-c342-47f7-82fd-385b1758527c`::stopping in state preparing (force False) >Thread-71845::DEBUG::2014-09-25 18:10:30,914::task::993::Storage.TaskManager.Task::(_decref) Task=`30ece177-c342-47f7-82fd-385b1758527c`::ref 1 aborting True >Thread-71845::INFO::2014-09-25 18:10:30,914::task::1171::Storage.TaskManager.Task::(prepare) Task=`30ece177-c342-47f7-82fd-385b1758527c`::aborting: Task is aborted: u'' - code 100 >Thread-71845::DEBUG::2014-09-25 18:10:30,914::task::1176::Storage.TaskManager.Task::(prepare) Task=`30ece177-c342-47f7-82fd-385b1758527c`::Prepare: aborted: >Thread-71845::DEBUG::2014-09-25 18:10:30,914::task::993::Storage.TaskManager.Task::(_decref) Task=`30ece177-c342-47f7-82fd-385b1758527c`::ref 0 aborting True >Thread-71845::DEBUG::2014-09-25 18:10:30,915::task::928::Storage.TaskManager.Task::(_doAbort) Task=`30ece177-c342-47f7-82fd-385b1758527c`::Task._doAbort: force False >Thread-71845::DEBUG::2014-09-25 18:10:30,915::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71845::DEBUG::2014-09-25 18:10:30,915::task::595::Storage.TaskManager.Task::(_updateState) Task=`30ece177-c342-47f7-82fd-385b1758527c`::moving from state preparing -> state aborting >Thread-71845::DEBUG::2014-09-25 18:10:30,915::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`30ece177-c342-47f7-82fd-385b1758527c`::_aborting: recover policy none >Thread-71845::DEBUG::2014-09-25 18:10:30,915::task::595::Storage.TaskManager.Task::(_updateState) Task=`30ece177-c342-47f7-82fd-385b1758527c`::moving from state aborting -> state failed >Thread-71845::DEBUG::2014-09-25 18:10:30,916::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71845::DEBUG::2014-09-25 18:10:30,916::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71845::ERROR::2014-09-25 18:10:30,916::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71845::DEBUG::2014-09-25 18:10:30,917::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:31,026::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:31,027::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71846::DEBUG::2014-09-25 18:10:31,029::task::595::Storage.TaskManager.Task::(_updateState) Task=`ff29f850-b8f1-4617-a980-865063a7a95d`::moving from state init -> state preparing >Thread-71846::INFO::2014-09-25 18:10:31,029::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71846::INFO::2014-09-25 18:10:31,030::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00277987', 'lastCheck': '6.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0039765', 'lastCheck': '3.0', 'valid': True}} >Thread-71846::DEBUG::2014-09-25 18:10:31,030::task::1191::Storage.TaskManager.Task::(prepare) Task=`ff29f850-b8f1-4617-a980-865063a7a95d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00277987', 'lastCheck': '6.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0039765', 'lastCheck': '3.0', 'valid': True}} >Thread-71846::DEBUG::2014-09-25 18:10:31,030::task::595::Storage.TaskManager.Task::(_updateState) Task=`ff29f850-b8f1-4617-a980-865063a7a95d`::moving from state preparing -> state finished >Thread-71846::DEBUG::2014-09-25 18:10:31,031::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71846::DEBUG::2014-09-25 18:10:31,031::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71846::DEBUG::2014-09-25 18:10:31,031::task::993::Storage.TaskManager.Task::(_decref) Task=`ff29f850-b8f1-4617-a980-865063a7a95d`::ref 0 aborting False >Thread-71846::DEBUG::2014-09-25 18:10:31,033::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:31,056::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:31,057::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71847::DEBUG::2014-09-25 18:10:31,059::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:32,270::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:32,300::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019217 s, 53.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:34,107::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:34,108::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71848::DEBUG::2014-09-25 18:10:34,110::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:34,320::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:34,347::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155466 s, 65.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:34,432::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:34,433::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:34,433::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:34,433::__init__::232::IOProcess::(_processLogs) (110676) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:34,435::__init__::232::IOProcess::(_processLogs) (110676) Queuing response >Thread-79::DEBUG::2014-09-25 18:10:34,435::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:10:34,446::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00133463 s, 421 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:34,447::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:34,448::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:34,448::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:34,448::__init__::232::IOProcess::(_processLogs) (110677) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:34,452::__init__::232::IOProcess::(_processLogs) (110677) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:34,453::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:34,454::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:34,454::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:34,454::__init__::232::IOProcess::(_processLogs) (110678) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:34,455::__init__::232::IOProcess::(_processLogs) (110678) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:34,455::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:34,455::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:34,456::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:34,456::__init__::232::IOProcess::(_processLogs) (110679) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:34,457::__init__::232::IOProcess::(_processLogs) (110679) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:34,457::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:34,457::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:34,458::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:34,459::__init__::232::IOProcess::(_processLogs) (110680) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:34,459::__init__::232::IOProcess::(_processLogs) (110680) Queuing response >Dummy-148::DEBUG::2014-09-25 18:10:36,367::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:36,393::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158113 s, 64.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:37,125::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:37,126::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71849::DEBUG::2014-09-25 18:10:37,128::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-71818::WARNING::2014-09-25 18:10:38,009::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:10:38,010::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:10:38,010::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-29::DEBUG::2014-09-25 18:10:38,057::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:38,058::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:38,058::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:38,058::__init__::232::IOProcess::(_processLogs) (110681) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:38,059::__init__::232::IOProcess::(_processLogs) (110681) Queuing response >Thread-81::DEBUG::2014-09-25 18:10:38,060::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:10:38,082::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00485146 s, 76.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:38,083::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:38,084::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:38,084::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:38,084::__init__::232::IOProcess::(_processLogs) (110682) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:38,089::__init__::232::IOProcess::(_processLogs) (110682) Queuing response >Dummy-148::DEBUG::2014-09-25 18:10:38,413::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:38,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0291088 s, 35.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:40,142::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:40,143::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71850::DEBUG::2014-09-25 18:10:40,145::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:40,475::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:40,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169373 s, 60.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:40,942::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:40,944::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71851::DEBUG::2014-09-25 18:10:40,944::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71851::DEBUG::2014-09-25 18:10:40,946::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::moving from state init -> state preparing >Thread-71851::INFO::2014-09-25 18:10:40,946::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71851::ERROR::2014-09-25 18:10:40,946::task::866::Storage.TaskManager.Task::(_setError) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71851::DEBUG::2014-09-25 18:10:40,947::task::885::Storage.TaskManager.Task::(_run) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::Task._run: bb7d9528-f143-40cd-9fc2-3be4603bec04 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71851::DEBUG::2014-09-25 18:10:40,947::task::1217::Storage.TaskManager.Task::(stop) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::stopping in state preparing (force False) >Thread-71851::DEBUG::2014-09-25 18:10:40,947::task::993::Storage.TaskManager.Task::(_decref) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::ref 1 aborting True >Thread-71851::INFO::2014-09-25 18:10:40,948::task::1171::Storage.TaskManager.Task::(prepare) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::aborting: Task is aborted: u'' - code 100 >Thread-71851::DEBUG::2014-09-25 18:10:40,948::task::1176::Storage.TaskManager.Task::(prepare) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::Prepare: aborted: >Thread-71851::DEBUG::2014-09-25 18:10:40,948::task::993::Storage.TaskManager.Task::(_decref) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::ref 0 aborting True >Thread-71851::DEBUG::2014-09-25 18:10:40,948::task::928::Storage.TaskManager.Task::(_doAbort) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::Task._doAbort: force False >Thread-71851::DEBUG::2014-09-25 18:10:40,949::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71851::DEBUG::2014-09-25 18:10:40,949::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::moving from state preparing -> state aborting >Thread-71851::DEBUG::2014-09-25 18:10:40,949::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::_aborting: recover policy none >Thread-71851::DEBUG::2014-09-25 18:10:40,949::task::595::Storage.TaskManager.Task::(_updateState) Task=`bb7d9528-f143-40cd-9fc2-3be4603bec04`::moving from state aborting -> state failed >Thread-71851::DEBUG::2014-09-25 18:10:40,949::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71851::DEBUG::2014-09-25 18:10:40,950::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71851::ERROR::2014-09-25 18:10:40,950::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71851::DEBUG::2014-09-25 18:10:40,950::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:42,523::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:42,556::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209676 s, 48.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:43,159::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:43,161::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71852::DEBUG::2014-09-25 18:10:43,163::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:10:44,459::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:44,460::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:44,460::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:44,460::__init__::232::IOProcess::(_processLogs) (110683) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:44,461::__init__::232::IOProcess::(_processLogs) (110683) Queuing response >Thread-79::DEBUG::2014-09-25 18:10:44,462::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:10:44,484::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00617417 s, 91.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:44,486::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:44,486::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:44,486::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:44,486::__init__::232::IOProcess::(_processLogs) (110684) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:44,488::__init__::232::IOProcess::(_processLogs) (110684) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:44,489::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:44,490::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:44,490::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:44,490::__init__::232::IOProcess::(_processLogs) (110685) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:44,490::__init__::232::IOProcess::(_processLogs) (110685) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:44,491::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:44,492::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:44,492::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:44,492::__init__::232::IOProcess::(_processLogs) (110686) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:44,492::__init__::232::IOProcess::(_processLogs) (110686) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:44,493::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:44,493::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:44,494::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:44,494::__init__::232::IOProcess::(_processLogs) (110687) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:44,494::__init__::232::IOProcess::(_processLogs) (110687) Queuing response >Dummy-148::DEBUG::2014-09-25 18:10:44,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:44,605::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174216 s, 58.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:46,179::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:46,182::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71853::DEBUG::2014-09-25 18:10:46,189::task::595::Storage.TaskManager.Task::(_updateState) Task=`7fe81606-0aca-4e8f-a8e7-3772b7f96f54`::moving from state init -> state preparing >Thread-71853::INFO::2014-09-25 18:10:46,190::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71853::INFO::2014-09-25 18:10:46,192::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00617417', 'lastCheck': '1.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00485146', 'lastCheck': '8.1', 'valid': True}} >Thread-71853::DEBUG::2014-09-25 18:10:46,193::task::1191::Storage.TaskManager.Task::(prepare) Task=`7fe81606-0aca-4e8f-a8e7-3772b7f96f54`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00617417', 'lastCheck': '1.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '8.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00485146', 'lastCheck': '8.1', 'valid': True}} >Thread-71853::DEBUG::2014-09-25 18:10:46,194::task::595::Storage.TaskManager.Task::(_updateState) Task=`7fe81606-0aca-4e8f-a8e7-3772b7f96f54`::moving from state preparing -> state finished >Thread-71853::DEBUG::2014-09-25 18:10:46,195::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71853::DEBUG::2014-09-25 18:10:46,196::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71853::DEBUG::2014-09-25 18:10:46,197::task::993::Storage.TaskManager.Task::(_decref) Task=`7fe81606-0aca-4e8f-a8e7-3772b7f96f54`::ref 0 aborting False >Thread-71853::DEBUG::2014-09-25 18:10:46,204::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:46,226::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:46,229::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71854::DEBUG::2014-09-25 18:10:46,234::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:46,625::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:46,651::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181663 s, 56.4 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 18:10:48,011::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-29::DEBUG::2014-09-25 18:10:48,015::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:48,016::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:48,016::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:48,016::__init__::232::IOProcess::(_processLogs) (110688) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:10:48,017::__init__::232::IOProcess::(_processLogs) (110688) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:48,019::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:48,019::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:48,020::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:48,020::__init__::232::IOProcess::(_processLogs) (110689) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:10:48,021::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:48,021::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:48,021::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:48,021::__init__::232::IOProcess::(_processLogs) (110690) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:10:48,022::__init__::232::IOProcess::(_processLogs) (110690) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:48,091::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:48,092::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:48,092::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:48,092::__init__::232::IOProcess::(_processLogs) (110691) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:48,094::__init__::232::IOProcess::(_processLogs) (110691) Queuing response >Thread-81::DEBUG::2014-09-25 18:10:48,094::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:10:48,103::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000416489 s, 891 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:48,105::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:48,106::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:48,107::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:48,107::__init__::232::IOProcess::(_processLogs) (110692) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:48,108::__init__::232::IOProcess::(_processLogs) (110692) Queuing response >Dummy-148::DEBUG::2014-09-25 18:10:48,772::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:48,800::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142784 s, 71.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:49,288::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:49,289::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71858::DEBUG::2014-09-25 18:10:49,291::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:50,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:50,850::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167818 s, 61.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:50,986::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71859::DEBUG::2014-09-25 18:10:50,987::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:10:50,988::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71859::DEBUG::2014-09-25 18:10:50,990::task::595::Storage.TaskManager.Task::(_updateState) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::moving from state init -> state preparing >Thread-71859::INFO::2014-09-25 18:10:50,990::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71859::ERROR::2014-09-25 18:10:50,991::task::866::Storage.TaskManager.Task::(_setError) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71859::DEBUG::2014-09-25 18:10:50,991::task::885::Storage.TaskManager.Task::(_run) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::Task._run: cf3282fc-91ad-437c-890a-95831a18ddd5 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71859::DEBUG::2014-09-25 18:10:50,992::task::1217::Storage.TaskManager.Task::(stop) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::stopping in state preparing (force False) >Thread-71859::DEBUG::2014-09-25 18:10:50,992::task::993::Storage.TaskManager.Task::(_decref) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::ref 1 aborting True >Thread-71859::INFO::2014-09-25 18:10:50,992::task::1171::Storage.TaskManager.Task::(prepare) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::aborting: Task is aborted: u'' - code 100 >Thread-71859::DEBUG::2014-09-25 18:10:50,992::task::1176::Storage.TaskManager.Task::(prepare) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::Prepare: aborted: >Thread-71859::DEBUG::2014-09-25 18:10:50,993::task::993::Storage.TaskManager.Task::(_decref) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::ref 0 aborting True >Thread-71859::DEBUG::2014-09-25 18:10:50,993::task::928::Storage.TaskManager.Task::(_doAbort) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::Task._doAbort: force False >Thread-71859::DEBUG::2014-09-25 18:10:50,993::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71859::DEBUG::2014-09-25 18:10:50,993::task::595::Storage.TaskManager.Task::(_updateState) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::moving from state preparing -> state aborting >Thread-71859::DEBUG::2014-09-25 18:10:50,993::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::_aborting: recover policy none >Thread-71859::DEBUG::2014-09-25 18:10:50,994::task::595::Storage.TaskManager.Task::(_updateState) Task=`cf3282fc-91ad-437c-890a-95831a18ddd5`::moving from state aborting -> state failed >Thread-71859::DEBUG::2014-09-25 18:10:50,994::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71859::DEBUG::2014-09-25 18:10:50,994::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71859::ERROR::2014-09-25 18:10:50,994::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71859::DEBUG::2014-09-25 18:10:50,995::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:52,306::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:52,307::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71860::DEBUG::2014-09-25 18:10:52,309::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:52,870::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:52,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0211428 s, 48.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:54,496::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:54,496::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:54,497::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:54,497::__init__::232::IOProcess::(_processLogs) (110693) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:54,499::__init__::232::IOProcess::(_processLogs) (110693) Queuing response >Thread-79::DEBUG::2014-09-25 18:10:54,500::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:10:54,509::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000431359 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:54,510::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:54,511::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:54,511::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:54,511::__init__::232::IOProcess::(_processLogs) (110694) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:54,512::__init__::232::IOProcess::(_processLogs) (110694) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:54,513::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:54,514::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:54,514::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:54,514::__init__::232::IOProcess::(_processLogs) (110695) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:54,514::__init__::232::IOProcess::(_processLogs) (110695) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:54,516::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:54,516::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:54,516::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:54,516::__init__::232::IOProcess::(_processLogs) (110696) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:54,517::__init__::232::IOProcess::(_processLogs) (110696) Queuing response >Thread-29::DEBUG::2014-09-25 18:10:54,518::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:54,518::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:54,518::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:54,518::__init__::232::IOProcess::(_processLogs) (110697) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:10:54,519::__init__::232::IOProcess::(_processLogs) (110697) Queuing response >Dummy-148::DEBUG::2014-09-25 18:10:54,922::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:54,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167407 s, 61.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:55,325::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:55,326::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71861::DEBUG::2014-09-25 18:10:55,328::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:56,969::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:57,001::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203231 s, 50.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:58,110::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:58,111::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-81::DEBUG::2014-09-25 18:10:58,112::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:10:58,112::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:58,121::__init__::232::IOProcess::(_processLogs) (110698) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:58,122::__init__::232::IOProcess::(_processLogs) (110698) Queuing response >Thread-81::DEBUG::2014-09-25 18:10:58,122::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000390284 s, 951 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:10:58,124::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:10:58,124::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:10:58,125::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:10:58,125::__init__::232::IOProcess::(_processLogs) (110699) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:10:58,125::__init__::232::IOProcess::(_processLogs) (110699) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:10:58,343::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:10:58,344::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71862::DEBUG::2014-09-25 18:10:58,347::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:10:59,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:10:59,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156119 s, 65.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:01,026::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71863::DEBUG::2014-09-25 18:11:01,027::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:11:01,027::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71863::DEBUG::2014-09-25 18:11:01,030::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::moving from state init -> state preparing >Thread-71863::INFO::2014-09-25 18:11:01,030::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71863::ERROR::2014-09-25 18:11:01,030::task::866::Storage.TaskManager.Task::(_setError) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71863::DEBUG::2014-09-25 18:11:01,031::task::885::Storage.TaskManager.Task::(_run) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::Task._run: 5e0a0cb8-2706-45c4-bc5b-de8e9c48807a (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71863::DEBUG::2014-09-25 18:11:01,031::task::1217::Storage.TaskManager.Task::(stop) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::stopping in state preparing (force False) >Thread-71863::DEBUG::2014-09-25 18:11:01,031::task::993::Storage.TaskManager.Task::(_decref) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::ref 1 aborting True >Thread-71863::INFO::2014-09-25 18:11:01,032::task::1171::Storage.TaskManager.Task::(prepare) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::aborting: Task is aborted: u'' - code 100 >Thread-71863::DEBUG::2014-09-25 18:11:01,032::task::1176::Storage.TaskManager.Task::(prepare) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::Prepare: aborted: >Thread-71863::DEBUG::2014-09-25 18:11:01,032::task::993::Storage.TaskManager.Task::(_decref) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::ref 0 aborting True >Thread-71863::DEBUG::2014-09-25 18:11:01,032::task::928::Storage.TaskManager.Task::(_doAbort) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::Task._doAbort: force False >Thread-71863::DEBUG::2014-09-25 18:11:01,032::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71863::DEBUG::2014-09-25 18:11:01,033::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::moving from state preparing -> state aborting >Thread-71863::DEBUG::2014-09-25 18:11:01,033::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::_aborting: recover policy none >Thread-71863::DEBUG::2014-09-25 18:11:01,033::task::595::Storage.TaskManager.Task::(_updateState) Task=`5e0a0cb8-2706-45c4-bc5b-de8e9c48807a`::moving from state aborting -> state failed >Thread-71863::DEBUG::2014-09-25 18:11:01,033::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71863::DEBUG::2014-09-25 18:11:01,033::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71863::ERROR::2014-09-25 18:11:01,034::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71863::DEBUG::2014-09-25 18:11:01,034::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:01,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:01,097::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017091 s, 59.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:01,363::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:01,364::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71864::DEBUG::2014-09-25 18:11:01,367::task::595::Storage.TaskManager.Task::(_updateState) Task=`5ae1a028-d812-4bfb-affa-ea44c9a3b04e`::moving from state init -> state preparing >Thread-71864::INFO::2014-09-25 18:11:01,367::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71864::INFO::2014-09-25 18:11:01,368::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000431359', 'lastCheck': '6.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.4', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000390284', 'lastCheck': '3.2', 'valid': True}} >Thread-71864::DEBUG::2014-09-25 18:11:01,368::task::1191::Storage.TaskManager.Task::(prepare) Task=`5ae1a028-d812-4bfb-affa-ea44c9a3b04e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000431359', 'lastCheck': '6.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '23.4', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000390284', 'lastCheck': '3.2', 'valid': True}} >Thread-71864::DEBUG::2014-09-25 18:11:01,368::task::595::Storage.TaskManager.Task::(_updateState) Task=`5ae1a028-d812-4bfb-affa-ea44c9a3b04e`::moving from state preparing -> state finished >Thread-71864::DEBUG::2014-09-25 18:11:01,369::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71864::DEBUG::2014-09-25 18:11:01,369::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71864::DEBUG::2014-09-25 18:11:01,369::task::993::Storage.TaskManager.Task::(_decref) Task=`5ae1a028-d812-4bfb-affa-ea44c9a3b04e`::ref 0 aborting False >Thread-71864::DEBUG::2014-09-25 18:11:01,371::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:01,389::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:01,390::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71865::DEBUG::2014-09-25 18:11:01,392::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:03,117::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:03,146::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170931 s, 59.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:04,447::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:04,448::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71866::DEBUG::2014-09-25 18:11:04,450::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:11:04,520::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:04,521::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:04,521::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:04,521::__init__::232::IOProcess::(_processLogs) (110700) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:04,523::__init__::232::IOProcess::(_processLogs) (110700) Queuing response >Thread-79::DEBUG::2014-09-25 18:11:04,524::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:11:04,541::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0050164 s, 112 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:04,543::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:04,543::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:04,543::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:04,544::__init__::232::IOProcess::(_processLogs) (110701) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:04,550::__init__::232::IOProcess::(_processLogs) (110701) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:04,551::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:04,551::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:04,551::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:04,552::__init__::232::IOProcess::(_processLogs) (110702) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:04,552::__init__::232::IOProcess::(_processLogs) (110702) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:04,553::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:04,553::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:04,553::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:04,553::__init__::232::IOProcess::(_processLogs) (110703) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:04,554::__init__::232::IOProcess::(_processLogs) (110703) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:04,554::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:04,555::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:04,555::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:04,555::__init__::232::IOProcess::(_processLogs) (110704) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:04,555::__init__::232::IOProcess::(_processLogs) (110704) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:05,166::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:05,199::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0224516 s, 45.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:11:07,219::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:07,249::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194759 s, 52.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:07,466::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:07,467::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71867::DEBUG::2014-09-25 18:11:07,469::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:11:08,127::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:08,128::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:08,128::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:08,128::__init__::232::IOProcess::(_processLogs) (110705) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:08,145::__init__::232::IOProcess::(_processLogs) (110705) Queuing response >Thread-81::DEBUG::2014-09-25 18:11:08,146::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:11:08,182::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00869179 s, 42.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:08,183::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:08,183::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:08,184::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:08,184::__init__::232::IOProcess::(_processLogs) (110706) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:08,189::__init__::232::IOProcess::(_processLogs) (110706) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:09,268::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:09,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169521 s, 60.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:10,483::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:10,485::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71868::DEBUG::2014-09-25 18:11:10,487::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:11,065::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71869::DEBUG::2014-09-25 18:11:11,066::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:11:11,066::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71869::DEBUG::2014-09-25 18:11:11,068::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::moving from state init -> state preparing >Thread-71869::INFO::2014-09-25 18:11:11,069::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71869::ERROR::2014-09-25 18:11:11,069::task::866::Storage.TaskManager.Task::(_setError) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71869::DEBUG::2014-09-25 18:11:11,070::task::885::Storage.TaskManager.Task::(_run) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::Task._run: dd0987c7-e138-4c3b-a34f-c0cad46243ed (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71869::DEBUG::2014-09-25 18:11:11,070::task::1217::Storage.TaskManager.Task::(stop) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::stopping in state preparing (force False) >Thread-71869::DEBUG::2014-09-25 18:11:11,070::task::993::Storage.TaskManager.Task::(_decref) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::ref 1 aborting True >Thread-71869::INFO::2014-09-25 18:11:11,070::task::1171::Storage.TaskManager.Task::(prepare) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::aborting: Task is aborted: u'' - code 100 >Thread-71869::DEBUG::2014-09-25 18:11:11,070::task::1176::Storage.TaskManager.Task::(prepare) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::Prepare: aborted: >Thread-71869::DEBUG::2014-09-25 18:11:11,071::task::993::Storage.TaskManager.Task::(_decref) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::ref 0 aborting True >Thread-71869::DEBUG::2014-09-25 18:11:11,071::task::928::Storage.TaskManager.Task::(_doAbort) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::Task._doAbort: force False >Thread-71869::DEBUG::2014-09-25 18:11:11,071::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71869::DEBUG::2014-09-25 18:11:11,071::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::moving from state preparing -> state aborting >Thread-71869::DEBUG::2014-09-25 18:11:11,072::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::_aborting: recover policy none >Thread-71869::DEBUG::2014-09-25 18:11:11,072::task::595::Storage.TaskManager.Task::(_updateState) Task=`dd0987c7-e138-4c3b-a34f-c0cad46243ed`::moving from state aborting -> state failed >Thread-71869::DEBUG::2014-09-25 18:11:11,072::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71869::DEBUG::2014-09-25 18:11:11,072::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71869::ERROR::2014-09-25 18:11:11,073::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71869::DEBUG::2014-09-25 18:11:11,073::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:11,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:11,345::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0194344 s, 52.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:11:13,365::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:13,398::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0250947 s, 40.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:13,502::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:13,503::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71870::DEBUG::2014-09-25 18:11:13,505::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:11:14,557::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:14,557::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:14,558::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:14,558::__init__::232::IOProcess::(_processLogs) (110707) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:14,563::__init__::232::IOProcess::(_processLogs) (110707) Queuing response >Thread-79::DEBUG::2014-09-25 18:11:14,564::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:11:14,578::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00252898 s, 222 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:14,579::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:14,580::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:14,580::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:14,580::__init__::232::IOProcess::(_processLogs) (110708) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:14,582::__init__::232::IOProcess::(_processLogs) (110708) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:14,584::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:14,584::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:14,585::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:14,585::__init__::232::IOProcess::(_processLogs) (110709) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:14,585::__init__::232::IOProcess::(_processLogs) (110709) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:14,586::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:14,586::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:14,587::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:14,587::__init__::232::IOProcess::(_processLogs) (110710) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:14,588::__init__::232::IOProcess::(_processLogs) (110710) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:14,588::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:14,589::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:14,590::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:14,590::__init__::232::IOProcess::(_processLogs) (110711) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:14,590::__init__::232::IOProcess::(_processLogs) (110711) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:15,418::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:15,460::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0284315 s, 36.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:16,521::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:16,524::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71871::DEBUG::2014-09-25 18:11:16,533::task::595::Storage.TaskManager.Task::(_updateState) Task=`ff1ebc5a-fce6-402e-9ba9-8ce8699e9a7b`::moving from state init -> state preparing >Thread-71871::INFO::2014-09-25 18:11:16,534::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71871::INFO::2014-09-25 18:11:16,536::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00252898', 'lastCheck': '1.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00869179', 'lastCheck': '8.3', 'valid': True}} >Thread-71871::DEBUG::2014-09-25 18:11:16,538::task::1191::Storage.TaskManager.Task::(prepare) Task=`ff1ebc5a-fce6-402e-9ba9-8ce8699e9a7b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00252898', 'lastCheck': '1.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '38.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00869179', 'lastCheck': '8.3', 'valid': True}} >Thread-71871::DEBUG::2014-09-25 18:11:16,539::task::595::Storage.TaskManager.Task::(_updateState) Task=`ff1ebc5a-fce6-402e-9ba9-8ce8699e9a7b`::moving from state preparing -> state finished >Thread-71871::DEBUG::2014-09-25 18:11:16,540::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71871::DEBUG::2014-09-25 18:11:16,540::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71871::DEBUG::2014-09-25 18:11:16,542::task::993::Storage.TaskManager.Task::(_decref) Task=`ff1ebc5a-fce6-402e-9ba9-8ce8699e9a7b`::ref 0 aborting False >Thread-71871::DEBUG::2014-09-25 18:11:16,550::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:16,568::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:16,571::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71872::DEBUG::2014-09-25 18:11:16,573::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:17,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:17,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134166 s, 76.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:18,191::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:18,192::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:18,192::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:18,192::__init__::232::IOProcess::(_processLogs) (110712) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:18,200::__init__::232::IOProcess::(_processLogs) (110712) Queuing response >Thread-81::DEBUG::2014-09-25 18:11:18,201::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:11:18,230::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0128332 s, 28.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:18,232::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:18,232::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:18,232::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:18,232::__init__::232::IOProcess::(_processLogs) (110713) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:18,239::__init__::232::IOProcess::(_processLogs) (110713) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:19,524::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:19,627::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:19,629::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71873::DEBUG::2014-09-25 18:11:19,631::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:19,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.2298 s, 4.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:21,097::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71874::DEBUG::2014-09-25 18:11:21,098::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:11:21,098::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71874::DEBUG::2014-09-25 18:11:21,100::task::595::Storage.TaskManager.Task::(_updateState) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::moving from state init -> state preparing >Thread-71874::INFO::2014-09-25 18:11:21,101::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71874::ERROR::2014-09-25 18:11:21,101::task::866::Storage.TaskManager.Task::(_setError) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71874::DEBUG::2014-09-25 18:11:21,102::task::885::Storage.TaskManager.Task::(_run) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::Task._run: dcca30e7-04a6-4c42-81db-9edac97ba90d (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71874::DEBUG::2014-09-25 18:11:21,102::task::1217::Storage.TaskManager.Task::(stop) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::stopping in state preparing (force False) >Thread-71874::DEBUG::2014-09-25 18:11:21,102::task::993::Storage.TaskManager.Task::(_decref) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::ref 1 aborting True >Thread-71874::INFO::2014-09-25 18:11:21,102::task::1171::Storage.TaskManager.Task::(prepare) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::aborting: Task is aborted: u'' - code 100 >Thread-71874::DEBUG::2014-09-25 18:11:21,102::task::1176::Storage.TaskManager.Task::(prepare) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::Prepare: aborted: >Thread-71874::DEBUG::2014-09-25 18:11:21,103::task::993::Storage.TaskManager.Task::(_decref) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::ref 0 aborting True >Thread-71874::DEBUG::2014-09-25 18:11:21,103::task::928::Storage.TaskManager.Task::(_doAbort) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::Task._doAbort: force False >Thread-71874::DEBUG::2014-09-25 18:11:21,103::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71874::DEBUG::2014-09-25 18:11:21,103::task::595::Storage.TaskManager.Task::(_updateState) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::moving from state preparing -> state aborting >Thread-71874::DEBUG::2014-09-25 18:11:21,104::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::_aborting: recover policy none >Thread-71874::DEBUG::2014-09-25 18:11:21,104::task::595::Storage.TaskManager.Task::(_updateState) Task=`dcca30e7-04a6-4c42-81db-9edac97ba90d`::moving from state aborting -> state failed >Thread-71874::DEBUG::2014-09-25 18:11:21,104::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71874::DEBUG::2014-09-25 18:11:21,104::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71874::ERROR::2014-09-25 18:11:21,104::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71874::DEBUG::2014-09-25 18:11:21,105::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:21,795::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:21,841::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0277378 s, 36.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:22,646::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:22,647::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71875::DEBUG::2014-09-25 18:11:22,649::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:23,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:23,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209442 s, 48.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:24,590::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:24,591::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:24,591::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:24,591::__init__::232::IOProcess::(_processLogs) (110714) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:24,602::__init__::232::IOProcess::(_processLogs) (110714) Queuing response >Thread-79::DEBUG::2014-09-25 18:11:24,604::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:11:24,613::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000377381 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:24,614::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:24,614::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:24,615::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:24,615::__init__::232::IOProcess::(_processLogs) (110715) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:24,615::__init__::232::IOProcess::(_processLogs) (110715) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:24,617::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:24,617::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:24,618::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:24,618::__init__::232::IOProcess::(_processLogs) (110716) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:24,618::__init__::232::IOProcess::(_processLogs) (110716) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:24,619::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:24,619::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:24,620::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:24,620::__init__::232::IOProcess::(_processLogs) (110717) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:24,621::__init__::232::IOProcess::(_processLogs) (110717) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:24,621::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:24,621::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:24,621::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:24,622::__init__::232::IOProcess::(_processLogs) (110718) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:24,622::__init__::232::IOProcess::(_processLogs) (110718) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:25,663::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:25,664::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71876::DEBUG::2014-09-25 18:11:25,667::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:11:25,882::__init__::232::IOProcess::(_processLogs) (110225) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:25,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:25,955::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0195161 s, 52.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:11:27,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:28,008::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0208667 s, 49.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:28,241::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:28,241::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:28,242::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:28,242::__init__::232::IOProcess::(_processLogs) (110719) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:28,242::__init__::232::IOProcess::(_processLogs) (110719) Queuing response >Thread-81::DEBUG::2014-09-25 18:11:28,244::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:11:28,252::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000342672 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:28,253::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:28,254::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:28,255::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:28,255::__init__::232::IOProcess::(_processLogs) (110720) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:28,256::__init__::232::IOProcess::(_processLogs) (110720) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:28,682::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:28,683::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71877::DEBUG::2014-09-25 18:11:28,685::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:11:28,891::__init__::232::IOProcess::(_processLogs) (110282) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:30,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:30,069::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0205498 s, 49.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:31,137::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71878::DEBUG::2014-09-25 18:11:31,138::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:11:31,138::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71878::DEBUG::2014-09-25 18:11:31,140::task::595::Storage.TaskManager.Task::(_updateState) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::moving from state init -> state preparing >Thread-71878::INFO::2014-09-25 18:11:31,141::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71878::ERROR::2014-09-25 18:11:31,141::task::866::Storage.TaskManager.Task::(_setError) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71878::DEBUG::2014-09-25 18:11:31,142::task::885::Storage.TaskManager.Task::(_run) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::Task._run: 766ecdfc-eb18-4133-9baa-ec7c2258c7f9 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71878::DEBUG::2014-09-25 18:11:31,142::task::1217::Storage.TaskManager.Task::(stop) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::stopping in state preparing (force False) >Thread-71878::DEBUG::2014-09-25 18:11:31,142::task::993::Storage.TaskManager.Task::(_decref) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::ref 1 aborting True >Thread-71878::INFO::2014-09-25 18:11:31,142::task::1171::Storage.TaskManager.Task::(prepare) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::aborting: Task is aborted: u'' - code 100 >Thread-71878::DEBUG::2014-09-25 18:11:31,142::task::1176::Storage.TaskManager.Task::(prepare) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::Prepare: aborted: >Thread-71878::DEBUG::2014-09-25 18:11:31,143::task::993::Storage.TaskManager.Task::(_decref) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::ref 0 aborting True >Thread-71878::DEBUG::2014-09-25 18:11:31,143::task::928::Storage.TaskManager.Task::(_doAbort) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::Task._doAbort: force False >Thread-71878::DEBUG::2014-09-25 18:11:31,143::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71878::DEBUG::2014-09-25 18:11:31,143::task::595::Storage.TaskManager.Task::(_updateState) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::moving from state preparing -> state aborting >Thread-71878::DEBUG::2014-09-25 18:11:31,144::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::_aborting: recover policy none >Thread-71878::DEBUG::2014-09-25 18:11:31,144::task::595::Storage.TaskManager.Task::(_updateState) Task=`766ecdfc-eb18-4133-9baa-ec7c2258c7f9`::moving from state aborting -> state failed >Thread-71878::DEBUG::2014-09-25 18:11:31,144::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71878::DEBUG::2014-09-25 18:11:31,144::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71878::ERROR::2014-09-25 18:11:31,144::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71878::DEBUG::2014-09-25 18:11:31,145::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:31,700::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:31,701::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71879::DEBUG::2014-09-25 18:11:31,703::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2dbbc83-09fb-4dc2-bc55-006834a81514`::moving from state init -> state preparing >Thread-71879::INFO::2014-09-25 18:11:31,703::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71879::INFO::2014-09-25 18:11:31,704::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000377381', 'lastCheck': '7.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000342672', 'lastCheck': '3.4', 'valid': True}} >Thread-71879::DEBUG::2014-09-25 18:11:31,704::task::1191::Storage.TaskManager.Task::(prepare) Task=`d2dbbc83-09fb-4dc2-bc55-006834a81514`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000377381', 'lastCheck': '7.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '53.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000342672', 'lastCheck': '3.4', 'valid': True}} >Thread-71879::DEBUG::2014-09-25 18:11:31,704::task::595::Storage.TaskManager.Task::(_updateState) Task=`d2dbbc83-09fb-4dc2-bc55-006834a81514`::moving from state preparing -> state finished >Thread-71879::DEBUG::2014-09-25 18:11:31,704::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71879::DEBUG::2014-09-25 18:11:31,705::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71879::DEBUG::2014-09-25 18:11:31,705::task::993::Storage.TaskManager.Task::(_decref) Task=`d2dbbc83-09fb-4dc2-bc55-006834a81514`::ref 0 aborting False >Thread-71879::DEBUG::2014-09-25 18:11:31,707::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:31,721::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:31,722::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71880::DEBUG::2014-09-25 18:11:31,724::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:32,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:32,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0218359 s, 46.9 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:11:34,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:34,183::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133888 s, 76.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:34,624::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:34,624::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:34,624::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:34,624::__init__::232::IOProcess::(_processLogs) (110721) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:34,628::__init__::232::IOProcess::(_processLogs) (110721) Queuing response >Thread-79::DEBUG::2014-09-25 18:11:34,628::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:11:34,645::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00199997 s, 281 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:34,647::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:34,647::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:34,647::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:34,648::__init__::232::IOProcess::(_processLogs) (110722) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:34,648::__init__::232::IOProcess::(_processLogs) (110722) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:34,650::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:34,650::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:34,651::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:34,651::__init__::232::IOProcess::(_processLogs) (110723) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:34,651::__init__::232::IOProcess::(_processLogs) (110723) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:34,652::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:34,652::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:34,653::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:34,653::__init__::232::IOProcess::(_processLogs) (110724) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:34,653::__init__::232::IOProcess::(_processLogs) (110724) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:34,654::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:34,654::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:34,655::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:34,656::__init__::232::IOProcess::(_processLogs) (110725) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:34,656::__init__::232::IOProcess::(_processLogs) (110725) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:34,778::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:34,779::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71881::DEBUG::2014-09-25 18:11:34,781::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:11:34,906::__init__::232::IOProcess::(_processLogs) (110332) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:36,203::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:36,239::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0250135 s, 40.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:37,797::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:37,798::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71882::DEBUG::2014-09-25 18:11:37,800::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:11:38,256::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:38,256::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:38,256::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:38,257::__init__::232::IOProcess::(_processLogs) (110726) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:38,257::__init__::232::IOProcess::(_processLogs) (110726) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:38,258::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-81::DEBUG::2014-09-25 18:11:38,259::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:11:38,279::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000269129 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:38,282::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:38,283::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:38,284::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:38,285::__init__::232::IOProcess::(_processLogs) (110727) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:38,286::__init__::232::IOProcess::(_processLogs) (110727) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:38,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0234277 s, 43.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:11:40,311::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:40,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015655 s, 65.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:40,823::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:40,825::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71883::DEBUG::2014-09-25 18:11:40,827::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:41,177::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71884::DEBUG::2014-09-25 18:11:41,178::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:11:41,178::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71884::DEBUG::2014-09-25 18:11:41,180::task::595::Storage.TaskManager.Task::(_updateState) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::moving from state init -> state preparing >Thread-71884::INFO::2014-09-25 18:11:41,181::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71884::ERROR::2014-09-25 18:11:41,181::task::866::Storage.TaskManager.Task::(_setError) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71884::DEBUG::2014-09-25 18:11:41,182::task::885::Storage.TaskManager.Task::(_run) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::Task._run: e41d6ebe-a359-4273-afaf-a1cc64f96b6d (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71884::DEBUG::2014-09-25 18:11:41,182::task::1217::Storage.TaskManager.Task::(stop) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::stopping in state preparing (force False) >Thread-71884::DEBUG::2014-09-25 18:11:41,182::task::993::Storage.TaskManager.Task::(_decref) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::ref 1 aborting True >Thread-71884::INFO::2014-09-25 18:11:41,182::task::1171::Storage.TaskManager.Task::(prepare) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::aborting: Task is aborted: u'' - code 100 >Thread-71884::DEBUG::2014-09-25 18:11:41,183::task::1176::Storage.TaskManager.Task::(prepare) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::Prepare: aborted: >Thread-71884::DEBUG::2014-09-25 18:11:41,183::task::993::Storage.TaskManager.Task::(_decref) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::ref 0 aborting True >Thread-71884::DEBUG::2014-09-25 18:11:41,183::task::928::Storage.TaskManager.Task::(_doAbort) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::Task._doAbort: force False >Thread-71884::DEBUG::2014-09-25 18:11:41,183::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71884::DEBUG::2014-09-25 18:11:41,184::task::595::Storage.TaskManager.Task::(_updateState) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::moving from state preparing -> state aborting >Thread-71884::DEBUG::2014-09-25 18:11:41,184::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::_aborting: recover policy none >Thread-71884::DEBUG::2014-09-25 18:11:41,184::task::595::Storage.TaskManager.Task::(_updateState) Task=`e41d6ebe-a359-4273-afaf-a1cc64f96b6d`::moving from state aborting -> state failed >Thread-71884::DEBUG::2014-09-25 18:11:41,184::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71884::DEBUG::2014-09-25 18:11:41,184::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71884::ERROR::2014-09-25 18:11:41,185::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71884::DEBUG::2014-09-25 18:11:41,185::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:42,358::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:42,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132592 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:43,845::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:43,846::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71885::DEBUG::2014-09-25 18:11:43,848::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:44,400::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:44,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160914 s, 63.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:44,657::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:44,657::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:44,657::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:44,657::__init__::232::IOProcess::(_processLogs) (110728) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:44,659::__init__::232::IOProcess::(_processLogs) (110728) Queuing response >Thread-79::DEBUG::2014-09-25 18:11:44,660::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:11:44,682::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00945508 s, 59.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:44,683::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:44,684::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:44,684::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:44,684::__init__::232::IOProcess::(_processLogs) (110729) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:44,687::__init__::232::IOProcess::(_processLogs) (110729) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:44,688::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:44,689::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:44,690::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:44,690::__init__::232::IOProcess::(_processLogs) (110730) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:44,690::__init__::232::IOProcess::(_processLogs) (110730) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:44,691::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:44,691::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:44,691::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:44,692::__init__::232::IOProcess::(_processLogs) (110731) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:44,692::__init__::232::IOProcess::(_processLogs) (110731) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:44,693::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:44,693::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:44,694::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:44,694::__init__::232::IOProcess::(_processLogs) (110732) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:44,695::__init__::232::IOProcess::(_processLogs) (110732) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:46,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:46,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0267291 s, 38.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:46,865::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:46,869::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71886::DEBUG::2014-09-25 18:11:46,876::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b36ce33-16b1-4e44-8fc1-f01a6a617db7`::moving from state init -> state preparing >Thread-71886::INFO::2014-09-25 18:11:46,877::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71886::INFO::2014-09-25 18:11:46,878::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00945508', 'lastCheck': '2.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000269129', 'lastCheck': '8.6', 'valid': True}} >Thread-71886::DEBUG::2014-09-25 18:11:46,879::task::1191::Storage.TaskManager.Task::(prepare) Task=`0b36ce33-16b1-4e44-8fc1-f01a6a617db7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00945508', 'lastCheck': '2.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '68.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000269129', 'lastCheck': '8.6', 'valid': True}} >Thread-71886::DEBUG::2014-09-25 18:11:46,880::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b36ce33-16b1-4e44-8fc1-f01a6a617db7`::moving from state preparing -> state finished >Thread-71886::DEBUG::2014-09-25 18:11:46,881::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71886::DEBUG::2014-09-25 18:11:46,882::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71886::DEBUG::2014-09-25 18:11:46,883::task::993::Storage.TaskManager.Task::(_decref) Task=`0b36ce33-16b1-4e44-8fc1-f01a6a617db7`::ref 0 aborting False >Thread-71886::DEBUG::2014-09-25 18:11:46,890::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:46,907::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:46,910::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71887::DEBUG::2014-09-25 18:11:46,914::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-71856::WARNING::2014-09-25 18:11:48,015::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:11:48,016::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:11:48,016::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-29::DEBUG::2014-09-25 18:11:48,290::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:48,290::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:48,290::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:48,291::__init__::232::IOProcess::(_processLogs) (110733) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:48,294::__init__::232::IOProcess::(_processLogs) (110733) Queuing response >Thread-81::DEBUG::2014-09-25 18:11:48,295::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:11:48,310::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00635035 s, 58.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:48,311::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:48,312::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:48,312::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:48,312::__init__::232::IOProcess::(_processLogs) (110734) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:48,318::__init__::232::IOProcess::(_processLogs) (110734) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:48,660::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:48,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132426 s, 77.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:49,959::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:49,961::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71888::DEBUG::2014-09-25 18:11:49,963::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:50,701::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:50,731::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186415 s, 54.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:51,213::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71889::DEBUG::2014-09-25 18:11:51,215::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:11:51,215::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71889::DEBUG::2014-09-25 18:11:51,217::task::595::Storage.TaskManager.Task::(_updateState) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::moving from state init -> state preparing >Thread-71889::INFO::2014-09-25 18:11:51,217::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71889::ERROR::2014-09-25 18:11:51,218::task::866::Storage.TaskManager.Task::(_setError) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71889::DEBUG::2014-09-25 18:11:51,218::task::885::Storage.TaskManager.Task::(_run) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::Task._run: 1f4eb433-6bee-481d-bc7d-e0de8dad1b80 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71889::DEBUG::2014-09-25 18:11:51,219::task::1217::Storage.TaskManager.Task::(stop) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::stopping in state preparing (force False) >Thread-71889::DEBUG::2014-09-25 18:11:51,219::task::993::Storage.TaskManager.Task::(_decref) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::ref 1 aborting True >Thread-71889::INFO::2014-09-25 18:11:51,219::task::1171::Storage.TaskManager.Task::(prepare) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::aborting: Task is aborted: u'' - code 100 >Thread-71889::DEBUG::2014-09-25 18:11:51,219::task::1176::Storage.TaskManager.Task::(prepare) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::Prepare: aborted: >Thread-71889::DEBUG::2014-09-25 18:11:51,220::task::993::Storage.TaskManager.Task::(_decref) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::ref 0 aborting True >Thread-71889::DEBUG::2014-09-25 18:11:51,220::task::928::Storage.TaskManager.Task::(_doAbort) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::Task._doAbort: force False >Thread-71889::DEBUG::2014-09-25 18:11:51,220::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71889::DEBUG::2014-09-25 18:11:51,220::task::595::Storage.TaskManager.Task::(_updateState) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::moving from state preparing -> state aborting >Thread-71889::DEBUG::2014-09-25 18:11:51,220::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::_aborting: recover policy none >Thread-71889::DEBUG::2014-09-25 18:11:51,221::task::595::Storage.TaskManager.Task::(_updateState) Task=`1f4eb433-6bee-481d-bc7d-e0de8dad1b80`::moving from state aborting -> state failed >Thread-71889::DEBUG::2014-09-25 18:11:51,221::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71889::DEBUG::2014-09-25 18:11:51,221::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71889::ERROR::2014-09-25 18:11:51,221::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71889::DEBUG::2014-09-25 18:11:51,222::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:52,751::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:52,776::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144717 s, 70.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:52,977::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:52,978::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71890::DEBUG::2014-09-25 18:11:52,980::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:11:54,695::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:54,696::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:54,696::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:54,696::__init__::232::IOProcess::(_processLogs) (110735) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:54,702::__init__::232::IOProcess::(_processLogs) (110735) Queuing response >Thread-79::DEBUG::2014-09-25 18:11:54,702::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:11:54,715::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00310936 s, 181 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:54,716::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:54,716::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:54,717::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:54,717::__init__::232::IOProcess::(_processLogs) (110736) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:54,719::__init__::232::IOProcess::(_processLogs) (110736) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:54,721::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:54,721::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:54,722::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:54,722::__init__::232::IOProcess::(_processLogs) (110737) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:54,728::__init__::232::IOProcess::(_processLogs) (110737) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:54,729::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:54,729::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:54,729::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:54,729::__init__::232::IOProcess::(_processLogs) (110738) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:54,736::__init__::232::IOProcess::(_processLogs) (110738) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:54,737::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:54,737::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:54,737::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:54,737::__init__::232::IOProcess::(_processLogs) (110739) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:11:54,740::__init__::232::IOProcess::(_processLogs) (110739) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:54,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:54,823::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167365 s, 61.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:56,004::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:56,006::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71891::DEBUG::2014-09-25 18:11:56,008::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:11:56,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:56,872::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165522 s, 61.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:58,021::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:58,022::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:58,022::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:58,022::__init__::232::IOProcess::(_processLogs) (110740) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:11:58,022::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:58,023::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:58,023::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:58,023::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:58,023::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:58,024::__init__::232::IOProcess::(_processLogs) (110741) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:11:58,024::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:58,024::__init__::232::IOProcess::(_processLogs) (110742) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:11:58,025::__init__::232::IOProcess::(_processLogs) (110740) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:58,027::__init__::232::IOProcess::(_processLogs) (110742) Queuing response >Thread-29::DEBUG::2014-09-25 18:11:58,320::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:58,320::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:58,321::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:58,322::__init__::232::IOProcess::(_processLogs) (110743) Got request for method 'statvfs' >Thread-81::DEBUG::2014-09-25 18:11:58,322::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:11:58,323::__init__::232::IOProcess::(_processLogs) (110743) Queuing response >Thread-81::DEBUG::2014-09-25 18:11:58,332::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000431302 s, 860 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:11:58,333::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:11:58,333::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:11:58,334::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:11:58,336::__init__::232::IOProcess::(_processLogs) (110744) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:11:58,336::__init__::232::IOProcess::(_processLogs) (110744) Queuing response >Dummy-148::DEBUG::2014-09-25 18:11:58,892::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:11:58,921::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160267 s, 63.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:11:59,021::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:11:59,022::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71895::DEBUG::2014-09-25 18:11:59,024::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:00,940::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:00,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0223296 s, 45.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:01,250::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71896::DEBUG::2014-09-25 18:12:01,251::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:12:01,251::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71896::DEBUG::2014-09-25 18:12:01,254::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::moving from state init -> state preparing >Thread-71896::INFO::2014-09-25 18:12:01,254::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71896::ERROR::2014-09-25 18:12:01,255::task::866::Storage.TaskManager.Task::(_setError) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71896::DEBUG::2014-09-25 18:12:01,255::task::885::Storage.TaskManager.Task::(_run) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::Task._run: 0f1eacd1-e997-4443-ab76-8bec4dca6c23 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71896::DEBUG::2014-09-25 18:12:01,255::task::1217::Storage.TaskManager.Task::(stop) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::stopping in state preparing (force False) >Thread-71896::DEBUG::2014-09-25 18:12:01,256::task::993::Storage.TaskManager.Task::(_decref) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::ref 1 aborting True >Thread-71896::INFO::2014-09-25 18:12:01,256::task::1171::Storage.TaskManager.Task::(prepare) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::aborting: Task is aborted: u'' - code 100 >Thread-71896::DEBUG::2014-09-25 18:12:01,256::task::1176::Storage.TaskManager.Task::(prepare) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::Prepare: aborted: >Thread-71896::DEBUG::2014-09-25 18:12:01,256::task::993::Storage.TaskManager.Task::(_decref) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::ref 0 aborting True >Thread-71896::DEBUG::2014-09-25 18:12:01,257::task::928::Storage.TaskManager.Task::(_doAbort) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::Task._doAbort: force False >Thread-71896::DEBUG::2014-09-25 18:12:01,257::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71896::DEBUG::2014-09-25 18:12:01,257::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::moving from state preparing -> state aborting >Thread-71896::DEBUG::2014-09-25 18:12:01,257::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::_aborting: recover policy none >Thread-71896::DEBUG::2014-09-25 18:12:01,257::task::595::Storage.TaskManager.Task::(_updateState) Task=`0f1eacd1-e997-4443-ab76-8bec4dca6c23`::moving from state aborting -> state failed >Thread-71896::DEBUG::2014-09-25 18:12:01,258::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71896::DEBUG::2014-09-25 18:12:01,258::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71896::ERROR::2014-09-25 18:12:01,258::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71896::DEBUG::2014-09-25 18:12:01,259::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:02,037::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:02,038::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71897::DEBUG::2014-09-25 18:12:02,040::task::595::Storage.TaskManager.Task::(_updateState) Task=`6cc6e641-471f-4c9b-9bc5-cb0c5adf578c`::moving from state init -> state preparing >Thread-71897::INFO::2014-09-25 18:12:02,041::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71897::INFO::2014-09-25 18:12:02,041::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00310936', 'lastCheck': '7.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '14.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000431302', 'lastCheck': '3.7', 'valid': True}} >Thread-71897::DEBUG::2014-09-25 18:12:02,042::task::1191::Storage.TaskManager.Task::(prepare) Task=`6cc6e641-471f-4c9b-9bc5-cb0c5adf578c`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00310936', 'lastCheck': '7.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '14.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000431302', 'lastCheck': '3.7', 'valid': True}} >Thread-71897::DEBUG::2014-09-25 18:12:02,042::task::595::Storage.TaskManager.Task::(_updateState) Task=`6cc6e641-471f-4c9b-9bc5-cb0c5adf578c`::moving from state preparing -> state finished >Thread-71897::DEBUG::2014-09-25 18:12:02,042::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71897::DEBUG::2014-09-25 18:12:02,042::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71897::DEBUG::2014-09-25 18:12:02,042::task::993::Storage.TaskManager.Task::(_decref) Task=`6cc6e641-471f-4c9b-9bc5-cb0c5adf578c`::ref 0 aborting False >Thread-71897::DEBUG::2014-09-25 18:12:02,045::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:02,062::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:02,063::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71898::DEBUG::2014-09-25 18:12:02,065::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:02,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:03,024::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153845 s, 66.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:04,742::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:04,742::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:04,743::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:04,743::__init__::232::IOProcess::(_processLogs) (110745) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:04,748::__init__::232::IOProcess::(_processLogs) (110745) Queuing response >Thread-79::DEBUG::2014-09-25 18:12:04,749::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:12:04,759::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00176128 s, 319 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:04,760::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:04,761::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:04,761::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:04,761::__init__::232::IOProcess::(_processLogs) (110746) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:04,767::__init__::232::IOProcess::(_processLogs) (110746) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:04,768::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:04,768::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:04,769::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:04,769::__init__::232::IOProcess::(_processLogs) (110747) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:04,770::__init__::232::IOProcess::(_processLogs) (110747) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:04,770::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:04,771::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:04,771::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:04,771::__init__::232::IOProcess::(_processLogs) (110748) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:04,771::__init__::232::IOProcess::(_processLogs) (110748) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:04,772::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:04,772::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:04,773::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:04,773::__init__::232::IOProcess::(_processLogs) (110749) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:04,773::__init__::232::IOProcess::(_processLogs) (110749) Queuing response >Dummy-148::DEBUG::2014-09-25 18:12:05,045::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:05,072::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.014906 s, 68.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:05,118::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:05,119::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71899::DEBUG::2014-09-25 18:12:05,121::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:07,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:07,121::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152706 s, 67.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:08,135::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:08,137::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71900::DEBUG::2014-09-25 18:12:08,139::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:12:08,336::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:08,336::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:08,337::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:08,337::__init__::232::IOProcess::(_processLogs) (110750) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:08,356::__init__::232::IOProcess::(_processLogs) (110750) Queuing response >Thread-81::DEBUG::2014-09-25 18:12:08,357::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:12:08,390::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00265808 s, 140 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:08,391::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:08,391::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:08,391::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:08,392::__init__::232::IOProcess::(_processLogs) (110751) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:08,398::__init__::232::IOProcess::(_processLogs) (110751) Queuing response >Dummy-148::DEBUG::2014-09-25 18:12:09,141::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:09,165::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151784 s, 67.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:11,154::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:11,156::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71901::DEBUG::2014-09-25 18:12:11,158::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:11,185::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:11,212::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0150751 s, 67.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:11,286::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71902::DEBUG::2014-09-25 18:12:11,287::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:12:11,287::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71902::DEBUG::2014-09-25 18:12:11,290::task::595::Storage.TaskManager.Task::(_updateState) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::moving from state init -> state preparing >Thread-71902::INFO::2014-09-25 18:12:11,290::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71902::ERROR::2014-09-25 18:12:11,290::task::866::Storage.TaskManager.Task::(_setError) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71902::DEBUG::2014-09-25 18:12:11,291::task::885::Storage.TaskManager.Task::(_run) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::Task._run: deb54702-007b-47f8-8b08-b82a3da09063 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71902::DEBUG::2014-09-25 18:12:11,291::task::1217::Storage.TaskManager.Task::(stop) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::stopping in state preparing (force False) >Thread-71902::DEBUG::2014-09-25 18:12:11,291::task::993::Storage.TaskManager.Task::(_decref) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::ref 1 aborting True >Thread-71902::INFO::2014-09-25 18:12:11,292::task::1171::Storage.TaskManager.Task::(prepare) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::aborting: Task is aborted: u'' - code 100 >Thread-71902::DEBUG::2014-09-25 18:12:11,292::task::1176::Storage.TaskManager.Task::(prepare) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::Prepare: aborted: >Thread-71902::DEBUG::2014-09-25 18:12:11,292::task::993::Storage.TaskManager.Task::(_decref) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::ref 0 aborting True >Thread-71902::DEBUG::2014-09-25 18:12:11,292::task::928::Storage.TaskManager.Task::(_doAbort) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::Task._doAbort: force False >Thread-71902::DEBUG::2014-09-25 18:12:11,293::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71902::DEBUG::2014-09-25 18:12:11,293::task::595::Storage.TaskManager.Task::(_updateState) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::moving from state preparing -> state aborting >Thread-71902::DEBUG::2014-09-25 18:12:11,293::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::_aborting: recover policy none >Thread-71902::DEBUG::2014-09-25 18:12:11,293::task::595::Storage.TaskManager.Task::(_updateState) Task=`deb54702-007b-47f8-8b08-b82a3da09063`::moving from state aborting -> state failed >Thread-71902::DEBUG::2014-09-25 18:12:11,293::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71902::DEBUG::2014-09-25 18:12:11,294::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71902::ERROR::2014-09-25 18:12:11,294::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71902::DEBUG::2014-09-25 18:12:11,294::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:13,232::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:13,253::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131893 s, 77.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:14,177::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:14,179::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71903::DEBUG::2014-09-25 18:12:14,180::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:12:14,775::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:14,775::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:14,775::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:14,776::__init__::232::IOProcess::(_processLogs) (110752) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:14,783::__init__::232::IOProcess::(_processLogs) (110752) Queuing response >Thread-79::DEBUG::2014-09-25 18:12:14,784::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:12:14,793::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000396038 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:14,794::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:14,795::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:14,795::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:14,795::__init__::232::IOProcess::(_processLogs) (110753) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:14,796::__init__::232::IOProcess::(_processLogs) (110753) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:14,797::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:14,797::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:14,798::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:14,798::__init__::232::IOProcess::(_processLogs) (110754) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:14,799::__init__::232::IOProcess::(_processLogs) (110754) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:14,799::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:14,800::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:14,800::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:14,801::__init__::232::IOProcess::(_processLogs) (110755) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:14,801::__init__::232::IOProcess::(_processLogs) (110755) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:14,801::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:14,802::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:14,802::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:14,803::__init__::232::IOProcess::(_processLogs) (110756) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:14,803::__init__::232::IOProcess::(_processLogs) (110756) Queuing response >Dummy-148::DEBUG::2014-09-25 18:12:15,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:15,304::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164459 s, 62.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:17,197::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:17,198::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71904::DEBUG::2014-09-25 18:12:17,201::task::595::Storage.TaskManager.Task::(_updateState) Task=`c0bb920b-cee6-4081-b547-45f7a520932b`::moving from state init -> state preparing >Thread-71904::INFO::2014-09-25 18:12:17,201::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71904::INFO::2014-09-25 18:12:17,202::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000396038', 'lastCheck': '2.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00265808', 'lastCheck': '8.8', 'valid': True}} >Thread-71904::DEBUG::2014-09-25 18:12:17,202::task::1191::Storage.TaskManager.Task::(prepare) Task=`c0bb920b-cee6-4081-b547-45f7a520932b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000396038', 'lastCheck': '2.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '29.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00265808', 'lastCheck': '8.8', 'valid': True}} >Thread-71904::DEBUG::2014-09-25 18:12:17,202::task::595::Storage.TaskManager.Task::(_updateState) Task=`c0bb920b-cee6-4081-b547-45f7a520932b`::moving from state preparing -> state finished >Thread-71904::DEBUG::2014-09-25 18:12:17,202::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71904::DEBUG::2014-09-25 18:12:17,203::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71904::DEBUG::2014-09-25 18:12:17,203::task::993::Storage.TaskManager.Task::(_decref) Task=`c0bb920b-cee6-4081-b547-45f7a520932b`::ref 0 aborting False >Thread-71904::DEBUG::2014-09-25 18:12:17,205::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:17,223::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:17,224::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71905::DEBUG::2014-09-25 18:12:17,226::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:17,324::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:17,351::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161986 s, 63.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:18,402::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:18,402::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:18,402::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:18,403::__init__::232::IOProcess::(_processLogs) (110757) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:18,412::__init__::232::IOProcess::(_processLogs) (110757) Queuing response >Thread-81::DEBUG::2014-09-25 18:12:18,414::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:12:18,433::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0057173 s, 64.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:18,435::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:18,435::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:18,435::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:18,436::__init__::232::IOProcess::(_processLogs) (110758) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:18,440::__init__::232::IOProcess::(_processLogs) (110758) Queuing response >Dummy-148::DEBUG::2014-09-25 18:12:19,371::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:19,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0202254 s, 50.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:20,269::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:20,270::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71906::DEBUG::2014-09-25 18:12:20,272::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:21,322::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71907::DEBUG::2014-09-25 18:12:21,323::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:12:21,323::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71907::DEBUG::2014-09-25 18:12:21,325::task::595::Storage.TaskManager.Task::(_updateState) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::moving from state init -> state preparing >Thread-71907::INFO::2014-09-25 18:12:21,326::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71907::ERROR::2014-09-25 18:12:21,326::task::866::Storage.TaskManager.Task::(_setError) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71907::DEBUG::2014-09-25 18:12:21,327::task::885::Storage.TaskManager.Task::(_run) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::Task._run: aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71907::DEBUG::2014-09-25 18:12:21,327::task::1217::Storage.TaskManager.Task::(stop) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::stopping in state preparing (force False) >Thread-71907::DEBUG::2014-09-25 18:12:21,327::task::993::Storage.TaskManager.Task::(_decref) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::ref 1 aborting True >Thread-71907::INFO::2014-09-25 18:12:21,327::task::1171::Storage.TaskManager.Task::(prepare) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::aborting: Task is aborted: u'' - code 100 >Thread-71907::DEBUG::2014-09-25 18:12:21,328::task::1176::Storage.TaskManager.Task::(prepare) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::Prepare: aborted: >Thread-71907::DEBUG::2014-09-25 18:12:21,328::task::993::Storage.TaskManager.Task::(_decref) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::ref 0 aborting True >Thread-71907::DEBUG::2014-09-25 18:12:21,328::task::928::Storage.TaskManager.Task::(_doAbort) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::Task._doAbort: force False >Thread-71907::DEBUG::2014-09-25 18:12:21,328::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71907::DEBUG::2014-09-25 18:12:21,328::task::595::Storage.TaskManager.Task::(_updateState) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::moving from state preparing -> state aborting >Thread-71907::DEBUG::2014-09-25 18:12:21,329::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::_aborting: recover policy none >Thread-71907::DEBUG::2014-09-25 18:12:21,329::task::595::Storage.TaskManager.Task::(_updateState) Task=`aee11ea2-c4f2-4cfa-8292-ef52fb3fcce0`::moving from state aborting -> state failed >Thread-71907::DEBUG::2014-09-25 18:12:21,329::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71907::DEBUG::2014-09-25 18:12:21,329::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71907::ERROR::2014-09-25 18:12:21,330::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71907::DEBUG::2014-09-25 18:12:21,330::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:21,427::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:21,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181938 s, 56.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:23,292::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:23,293::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71908::DEBUG::2014-09-25 18:12:23,295::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:23,477::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:23,532::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0339332 s, 30.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:24,804::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:24,804::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:24,805::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:24,805::__init__::232::IOProcess::(_processLogs) (110759) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:24,807::__init__::232::IOProcess::(_processLogs) (110759) Queuing response >Thread-79::DEBUG::2014-09-25 18:12:24,809::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:12:24,824::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00565971 s, 99.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:24,825::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:24,825::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:24,826::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:24,826::__init__::232::IOProcess::(_processLogs) (110760) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:24,827::__init__::232::IOProcess::(_processLogs) (110760) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:24,829::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:24,829::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:24,830::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:24,830::__init__::232::IOProcess::(_processLogs) (110761) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:24,830::__init__::232::IOProcess::(_processLogs) (110761) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:24,831::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:24,831::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:24,832::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:24,832::__init__::232::IOProcess::(_processLogs) (110762) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:24,832::__init__::232::IOProcess::(_processLogs) (110762) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:24,833::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:24,833::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:24,834::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:24,835::__init__::232::IOProcess::(_processLogs) (110763) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:24,835::__init__::232::IOProcess::(_processLogs) (110763) Queuing response >Dummy-148::DEBUG::2014-09-25 18:12:25,552::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:25,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145938 s, 70.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:26,311::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:26,313::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71909::DEBUG::2014-09-25 18:12:26,315::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:27,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:27,623::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018187 s, 56.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:28,441::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:28,442::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:28,442::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:28,442::__init__::232::IOProcess::(_processLogs) (110764) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:28,446::__init__::232::IOProcess::(_processLogs) (110764) Queuing response >Thread-81::DEBUG::2014-09-25 18:12:28,447::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:12:28,467::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00450191 s, 82.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:28,468::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:28,468::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:28,468::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:28,469::__init__::232::IOProcess::(_processLogs) (110765) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:28,471::__init__::232::IOProcess::(_processLogs) (110765) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:29,329::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:29,331::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71910::DEBUG::2014-09-25 18:12:29,333::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:29,643::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:29,678::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0226284 s, 45.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:31,359::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:31,361::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71911::DEBUG::2014-09-25 18:12:31,361::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71911::DEBUG::2014-09-25 18:12:31,363::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::moving from state init -> state preparing >Thread-71911::INFO::2014-09-25 18:12:31,363::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71911::ERROR::2014-09-25 18:12:31,364::task::866::Storage.TaskManager.Task::(_setError) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71911::DEBUG::2014-09-25 18:12:31,364::task::885::Storage.TaskManager.Task::(_run) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::Task._run: 2c82c50a-ddb5-4476-ab88-f0e119c42fc0 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71911::DEBUG::2014-09-25 18:12:31,364::task::1217::Storage.TaskManager.Task::(stop) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::stopping in state preparing (force False) >Thread-71911::DEBUG::2014-09-25 18:12:31,365::task::993::Storage.TaskManager.Task::(_decref) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::ref 1 aborting True >Thread-71911::INFO::2014-09-25 18:12:31,365::task::1171::Storage.TaskManager.Task::(prepare) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::aborting: Task is aborted: u'' - code 100 >Thread-71911::DEBUG::2014-09-25 18:12:31,365::task::1176::Storage.TaskManager.Task::(prepare) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::Prepare: aborted: >Thread-71911::DEBUG::2014-09-25 18:12:31,365::task::993::Storage.TaskManager.Task::(_decref) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::ref 0 aborting True >Thread-71911::DEBUG::2014-09-25 18:12:31,366::task::928::Storage.TaskManager.Task::(_doAbort) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::Task._doAbort: force False >Thread-71911::DEBUG::2014-09-25 18:12:31,366::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71911::DEBUG::2014-09-25 18:12:31,366::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::moving from state preparing -> state aborting >Thread-71911::DEBUG::2014-09-25 18:12:31,366::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::_aborting: recover policy none >Thread-71911::DEBUG::2014-09-25 18:12:31,366::task::595::Storage.TaskManager.Task::(_updateState) Task=`2c82c50a-ddb5-4476-ab88-f0e119c42fc0`::moving from state aborting -> state failed >Thread-71911::DEBUG::2014-09-25 18:12:31,367::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71911::DEBUG::2014-09-25 18:12:31,367::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71911::ERROR::2014-09-25 18:12:31,367::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71911::DEBUG::2014-09-25 18:12:31,368::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:31,697::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:31,730::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016373 s, 62.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:32,348::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:32,351::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71912::DEBUG::2014-09-25 18:12:32,358::task::595::Storage.TaskManager.Task::(_updateState) Task=`7674ff3d-5c4a-4acc-9248-f69595a906da`::moving from state init -> state preparing >Thread-71912::INFO::2014-09-25 18:12:32,359::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71912::INFO::2014-09-25 18:12:32,361::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00565971', 'lastCheck': '7.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00450191', 'lastCheck': '3.9', 'valid': True}} >Thread-71912::DEBUG::2014-09-25 18:12:32,362::task::1191::Storage.TaskManager.Task::(prepare) Task=`7674ff3d-5c4a-4acc-9248-f69595a906da`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00565971', 'lastCheck': '7.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '44.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00450191', 'lastCheck': '3.9', 'valid': True}} >Thread-71912::DEBUG::2014-09-25 18:12:32,362::task::595::Storage.TaskManager.Task::(_updateState) Task=`7674ff3d-5c4a-4acc-9248-f69595a906da`::moving from state preparing -> state finished >Thread-71912::DEBUG::2014-09-25 18:12:32,362::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71912::DEBUG::2014-09-25 18:12:32,362::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71912::DEBUG::2014-09-25 18:12:32,362::task::993::Storage.TaskManager.Task::(_decref) Task=`7674ff3d-5c4a-4acc-9248-f69595a906da`::ref 0 aborting False >Thread-71912::DEBUG::2014-09-25 18:12:32,365::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:32,391::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:32,391::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71913::DEBUG::2014-09-25 18:12:32,393::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:33,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:33,779::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180202 s, 56.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:34,835::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:34,835::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:34,836::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:34,836::__init__::232::IOProcess::(_processLogs) (110766) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:34,837::__init__::232::IOProcess::(_processLogs) (110766) Queuing response >Thread-79::DEBUG::2014-09-25 18:12:34,838::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:12:34,854::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00365697 s, 154 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:34,855::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:34,856::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:34,856::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:34,856::__init__::232::IOProcess::(_processLogs) (110767) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:34,858::__init__::232::IOProcess::(_processLogs) (110767) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:34,860::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:34,860::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:34,861::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:34,861::__init__::232::IOProcess::(_processLogs) (110768) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:34,861::__init__::232::IOProcess::(_processLogs) (110768) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:34,862::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:34,862::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:34,863::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:34,863::__init__::232::IOProcess::(_processLogs) (110769) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:34,863::__init__::232::IOProcess::(_processLogs) (110769) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:34,864::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:34,864::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:34,865::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:34,865::__init__::232::IOProcess::(_processLogs) (110770) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:34,866::__init__::232::IOProcess::(_processLogs) (110770) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:35,440::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:35,442::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71914::DEBUG::2014-09-25 18:12:35,444::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:35,798::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:35,829::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166142 s, 61.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:12:37,849::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:37,874::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154816 s, 66.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:38,457::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:38,459::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71915::DEBUG::2014-09-25 18:12:38,461::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 18:12:38,473::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 18:12:38,476::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:38,477::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:38,477::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:38,477::__init__::232::IOProcess::(_processLogs) (110771) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:12:38,478::__init__::232::IOProcess::(_processLogs) (110771) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:38,480::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:38,482::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:38,483::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:38,485::__init__::232::IOProcess::(_processLogs) (110772) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:12:38,487::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:38,487::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:38,489::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:38,490::__init__::232::IOProcess::(_processLogs) (110773) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:12:38,492::__init__::232::IOProcess::(_processLogs) (110773) Queuing response >Thread-81::DEBUG::2014-09-25 18:12:38,494::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 18:12:38,495::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:38,495::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:38,495::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:38,495::__init__::232::IOProcess::(_processLogs) (110774) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 18:12:38,518::__init__::232::IOProcess::(_processLogs) (110774) Queuing response >Thread-81::DEBUG::2014-09-25 18:12:38,519::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:12:38,520::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:38,520::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:38,520::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:38,520::__init__::232::IOProcess::(_processLogs) (110775) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 18:12:38,521::__init__::232::IOProcess::(_processLogs) (110775) Queuing response >Thread-81::DEBUG::2014-09-25 18:12:38,521::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 18:12:38,523::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:38,523::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:38,524::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:38,524::__init__::232::IOProcess::(_processLogs) (110776) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:38,524::__init__::232::IOProcess::(_processLogs) (110776) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:38,525::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:38,525::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:38,525::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:38,526::__init__::232::IOProcess::(_processLogs) (110777) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:12:38,526::__init__::232::IOProcess::(_processLogs) (110777) Queuing response >Thread-81::DEBUG::2014-09-25 18:12:38,526::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 18:12:38,527::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 18:12:38,527::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 18:12:38,528::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:38,528::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:38,529::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:38,529::__init__::232::IOProcess::(_processLogs) (110778) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:38,529::__init__::232::IOProcess::(_processLogs) (110778) Queuing response >Thread-81::DEBUG::2014-09-25 18:12:38,530::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:12:38,538::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000335366 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:38,540::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:38,540::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:38,540::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:38,540::__init__::232::IOProcess::(_processLogs) (110779) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:38,541::__init__::232::IOProcess::(_processLogs) (110779) Queuing response >Dummy-148::DEBUG::2014-09-25 18:12:39,894::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:39,917::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133842 s, 76.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:41,392::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:41,393::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71919::DEBUG::2014-09-25 18:12:41,393::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71919::DEBUG::2014-09-25 18:12:41,395::task::595::Storage.TaskManager.Task::(_updateState) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::moving from state init -> state preparing >Thread-71919::INFO::2014-09-25 18:12:41,396::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71919::ERROR::2014-09-25 18:12:41,396::task::866::Storage.TaskManager.Task::(_setError) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71919::DEBUG::2014-09-25 18:12:41,397::task::885::Storage.TaskManager.Task::(_run) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::Task._run: c85bb901-2ea3-4656-95ad-e442ed5b26c6 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71919::DEBUG::2014-09-25 18:12:41,397::task::1217::Storage.TaskManager.Task::(stop) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::stopping in state preparing (force False) >Thread-71919::DEBUG::2014-09-25 18:12:41,397::task::993::Storage.TaskManager.Task::(_decref) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::ref 1 aborting True >Thread-71919::INFO::2014-09-25 18:12:41,397::task::1171::Storage.TaskManager.Task::(prepare) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::aborting: Task is aborted: u'' - code 100 >Thread-71919::DEBUG::2014-09-25 18:12:41,398::task::1176::Storage.TaskManager.Task::(prepare) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::Prepare: aborted: >Thread-71919::DEBUG::2014-09-25 18:12:41,398::task::993::Storage.TaskManager.Task::(_decref) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::ref 0 aborting True >Thread-71919::DEBUG::2014-09-25 18:12:41,398::task::928::Storage.TaskManager.Task::(_doAbort) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::Task._doAbort: force False >Thread-71919::DEBUG::2014-09-25 18:12:41,398::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71919::DEBUG::2014-09-25 18:12:41,398::task::595::Storage.TaskManager.Task::(_updateState) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::moving from state preparing -> state aborting >Thread-71919::DEBUG::2014-09-25 18:12:41,399::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::_aborting: recover policy none >Thread-71919::DEBUG::2014-09-25 18:12:41,399::task::595::Storage.TaskManager.Task::(_updateState) Task=`c85bb901-2ea3-4656-95ad-e442ed5b26c6`::moving from state aborting -> state failed >Thread-71919::DEBUG::2014-09-25 18:12:41,399::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71919::DEBUG::2014-09-25 18:12:41,400::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71919::ERROR::2014-09-25 18:12:41,402::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71919::DEBUG::2014-09-25 18:12:41,403::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:41,477::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:41,478::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71920::DEBUG::2014-09-25 18:12:41,480::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:41,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:41,958::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133684 s, 76.6 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:12:43,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:44,009::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152953 s, 66.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:44,498::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:44,500::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71921::DEBUG::2014-09-25 18:12:44,502::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:12:44,866::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:44,866::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:44,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:44,867::__init__::232::IOProcess::(_processLogs) (110780) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:44,868::__init__::232::IOProcess::(_processLogs) (110780) Queuing response >Thread-79::DEBUG::2014-09-25 18:12:44,870::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:12:44,886::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00527854 s, 106 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:44,887::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:44,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:44,888::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:44,888::__init__::232::IOProcess::(_processLogs) (110781) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:44,889::__init__::232::IOProcess::(_processLogs) (110781) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:44,890::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:44,891::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:44,892::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:44,892::__init__::232::IOProcess::(_processLogs) (110782) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:44,892::__init__::232::IOProcess::(_processLogs) (110782) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:44,893::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:44,893::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:44,894::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:44,894::__init__::232::IOProcess::(_processLogs) (110783) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:44,894::__init__::232::IOProcess::(_processLogs) (110783) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:44,895::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:44,895::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:44,896::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:44,896::__init__::232::IOProcess::(_processLogs) (110784) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:44,897::__init__::232::IOProcess::(_processLogs) (110784) Queuing response >Dummy-148::DEBUG::2014-09-25 18:12:46,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:46,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0293883 s, 34.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:47,515::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:47,516::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71922::DEBUG::2014-09-25 18:12:47,518::task::595::Storage.TaskManager.Task::(_updateState) Task=`cc4546a7-2d42-4679-af9e-77355b1477d7`::moving from state init -> state preparing >Thread-71922::INFO::2014-09-25 18:12:47,519::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71922::INFO::2014-09-25 18:12:47,519::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00527854', 'lastCheck': '2.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000335366', 'lastCheck': '9.0', 'valid': True}} >Thread-71922::DEBUG::2014-09-25 18:12:47,519::task::1191::Storage.TaskManager.Task::(prepare) Task=`cc4546a7-2d42-4679-af9e-77355b1477d7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00527854', 'lastCheck': '2.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '59.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000335366', 'lastCheck': '9.0', 'valid': True}} >Thread-71922::DEBUG::2014-09-25 18:12:47,520::task::595::Storage.TaskManager.Task::(_updateState) Task=`cc4546a7-2d42-4679-af9e-77355b1477d7`::moving from state preparing -> state finished >Thread-71922::DEBUG::2014-09-25 18:12:47,520::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71922::DEBUG::2014-09-25 18:12:47,520::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71922::DEBUG::2014-09-25 18:12:47,520::task::993::Storage.TaskManager.Task::(_decref) Task=`cc4546a7-2d42-4679-af9e-77355b1477d7`::ref 0 aborting False >Thread-71922::DEBUG::2014-09-25 18:12:47,522::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:47,540::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:47,541::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71923::DEBUG::2014-09-25 18:12:47,543::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:48,280::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:48,308::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164214 s, 62.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:48,543::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:48,543::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:48,544::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:48,544::__init__::232::IOProcess::(_processLogs) (110785) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:48,546::__init__::232::IOProcess::(_processLogs) (110785) Queuing response >Thread-81::DEBUG::2014-09-25 18:12:48,547::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:12:48,561::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00473708 s, 78.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:48,563::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:48,563::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:48,563::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:48,563::__init__::232::IOProcess::(_processLogs) (110786) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:48,571::__init__::232::IOProcess::(_processLogs) (110786) Queuing response >Dummy-148::DEBUG::2014-09-25 18:12:50,328::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:50,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146193 s, 70.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:50,595::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:50,596::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71924::DEBUG::2014-09-25 18:12:50,598::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:51,429::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:51,429::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71925::DEBUG::2014-09-25 18:12:51,430::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71925::DEBUG::2014-09-25 18:12:51,431::task::595::Storage.TaskManager.Task::(_updateState) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::moving from state init -> state preparing >Thread-71925::INFO::2014-09-25 18:12:51,432::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71925::ERROR::2014-09-25 18:12:51,432::task::866::Storage.TaskManager.Task::(_setError) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71925::DEBUG::2014-09-25 18:12:51,433::task::885::Storage.TaskManager.Task::(_run) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::Task._run: a741c5d3-b9eb-4c4c-9567-d836d82db034 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71925::DEBUG::2014-09-25 18:12:51,433::task::1217::Storage.TaskManager.Task::(stop) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::stopping in state preparing (force False) >Thread-71925::DEBUG::2014-09-25 18:12:51,433::task::993::Storage.TaskManager.Task::(_decref) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::ref 1 aborting True >Thread-71925::INFO::2014-09-25 18:12:51,433::task::1171::Storage.TaskManager.Task::(prepare) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::aborting: Task is aborted: u'' - code 100 >Thread-71925::DEBUG::2014-09-25 18:12:51,434::task::1176::Storage.TaskManager.Task::(prepare) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::Prepare: aborted: >Thread-71925::DEBUG::2014-09-25 18:12:51,434::task::993::Storage.TaskManager.Task::(_decref) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::ref 0 aborting True >Thread-71925::DEBUG::2014-09-25 18:12:51,434::task::928::Storage.TaskManager.Task::(_doAbort) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::Task._doAbort: force False >Thread-71925::DEBUG::2014-09-25 18:12:51,434::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71925::DEBUG::2014-09-25 18:12:51,435::task::595::Storage.TaskManager.Task::(_updateState) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::moving from state preparing -> state aborting >Thread-71925::DEBUG::2014-09-25 18:12:51,435::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::_aborting: recover policy none >Thread-71925::DEBUG::2014-09-25 18:12:51,435::task::595::Storage.TaskManager.Task::(_updateState) Task=`a741c5d3-b9eb-4c4c-9567-d836d82db034`::moving from state aborting -> state failed >Thread-71925::DEBUG::2014-09-25 18:12:51,435::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71925::DEBUG::2014-09-25 18:12:51,435::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71925::ERROR::2014-09-25 18:12:51,436::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71925::DEBUG::2014-09-25 18:12:51,436::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:52,376::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:52,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0269309 s, 38.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:53,616::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:53,618::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71926::DEBUG::2014-09-25 18:12:53,619::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:54,558::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:54,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0197635 s, 51.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:54,897::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:54,897::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:54,897::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:54,898::__init__::232::IOProcess::(_processLogs) (110787) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:54,898::__init__::232::IOProcess::(_processLogs) (110787) Queuing response >Thread-79::DEBUG::2014-09-25 18:12:54,900::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:12:54,913::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00135132 s, 416 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:54,915::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:54,915::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:54,915::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:54,915::__init__::232::IOProcess::(_processLogs) (110788) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:54,918::__init__::232::IOProcess::(_processLogs) (110788) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:54,919::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:54,919::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:54,919::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:54,920::__init__::232::IOProcess::(_processLogs) (110789) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:54,921::__init__::232::IOProcess::(_processLogs) (110789) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:54,922::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:54,922::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:54,923::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:54,923::__init__::232::IOProcess::(_processLogs) (110790) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:54,927::__init__::232::IOProcess::(_processLogs) (110790) Queuing response >Thread-29::DEBUG::2014-09-25 18:12:54,928::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:54,928::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:54,928::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:54,928::__init__::232::IOProcess::(_processLogs) (110791) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:12:54,930::__init__::232::IOProcess::(_processLogs) (110791) Queuing response >Dummy-148::DEBUG::2014-09-25 18:12:56,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:56,633::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:56,637::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71927::DEBUG::2014-09-25 18:12:56,644::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:12:56,649::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0270807 s, 37.8 MB/s\n'; <rc> = 0 >Thread-71893::WARNING::2014-09-25 18:12:58,020::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:12:58,021::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:12:58,022::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-29::DEBUG::2014-09-25 18:12:58,573::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:58,573::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:58,573::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:58,573::__init__::232::IOProcess::(_processLogs) (110792) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:58,575::__init__::232::IOProcess::(_processLogs) (110792) Queuing response >Thread-81::DEBUG::2014-09-25 18:12:58,576::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:12:58,597::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00416467 s, 89.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:12:58,598::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:12:58,599::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:12:58,599::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:12:58,599::__init__::232::IOProcess::(_processLogs) (110793) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:12:58,601::__init__::232::IOProcess::(_processLogs) (110793) Queuing response >Dummy-148::DEBUG::2014-09-25 18:12:58,668::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:12:58,698::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168679 s, 60.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:12:59,660::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:12:59,661::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71928::DEBUG::2014-09-25 18:12:59,663::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:00,718::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:00,745::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155599 s, 65.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:01,461::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:01,462::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71929::DEBUG::2014-09-25 18:13:01,462::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71929::DEBUG::2014-09-25 18:13:01,464::task::595::Storage.TaskManager.Task::(_updateState) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::moving from state init -> state preparing >Thread-71929::INFO::2014-09-25 18:13:01,464::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71929::ERROR::2014-09-25 18:13:01,465::task::866::Storage.TaskManager.Task::(_setError) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71929::DEBUG::2014-09-25 18:13:01,465::task::885::Storage.TaskManager.Task::(_run) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::Task._run: 2276c259-f382-4a23-bc19-ba37d817181b (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71929::DEBUG::2014-09-25 18:13:01,465::task::1217::Storage.TaskManager.Task::(stop) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::stopping in state preparing (force False) >Thread-71929::DEBUG::2014-09-25 18:13:01,466::task::993::Storage.TaskManager.Task::(_decref) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::ref 1 aborting True >Thread-71929::INFO::2014-09-25 18:13:01,466::task::1171::Storage.TaskManager.Task::(prepare) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::aborting: Task is aborted: u'' - code 100 >Thread-71929::DEBUG::2014-09-25 18:13:01,466::task::1176::Storage.TaskManager.Task::(prepare) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::Prepare: aborted: >Thread-71929::DEBUG::2014-09-25 18:13:01,466::task::993::Storage.TaskManager.Task::(_decref) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::ref 0 aborting True >Thread-71929::DEBUG::2014-09-25 18:13:01,467::task::928::Storage.TaskManager.Task::(_doAbort) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::Task._doAbort: force False >Thread-71929::DEBUG::2014-09-25 18:13:01,467::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71929::DEBUG::2014-09-25 18:13:01,467::task::595::Storage.TaskManager.Task::(_updateState) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::moving from state preparing -> state aborting >Thread-71929::DEBUG::2014-09-25 18:13:01,467::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::_aborting: recover policy none >Thread-71929::DEBUG::2014-09-25 18:13:01,467::task::595::Storage.TaskManager.Task::(_updateState) Task=`2276c259-f382-4a23-bc19-ba37d817181b`::moving from state aborting -> state failed >Thread-71929::DEBUG::2014-09-25 18:13:01,468::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71929::DEBUG::2014-09-25 18:13:01,468::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71929::ERROR::2014-09-25 18:13:01,468::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71929::DEBUG::2014-09-25 18:13:01,468::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:02,681::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:02,683::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71930::DEBUG::2014-09-25 18:13:02,685::task::595::Storage.TaskManager.Task::(_updateState) Task=`3bc75749-9349-407e-a00f-191a6db8450b`::moving from state init -> state preparing >Thread-71930::INFO::2014-09-25 18:13:02,685::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71930::INFO::2014-09-25 18:13:02,686::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00135132', 'lastCheck': '7.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00416467', 'lastCheck': '4.1', 'valid': True}} >Thread-71930::DEBUG::2014-09-25 18:13:02,686::task::1191::Storage.TaskManager.Task::(prepare) Task=`3bc75749-9349-407e-a00f-191a6db8450b`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00135132', 'lastCheck': '7.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '4.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00416467', 'lastCheck': '4.1', 'valid': True}} >Thread-71930::DEBUG::2014-09-25 18:13:02,686::task::595::Storage.TaskManager.Task::(_updateState) Task=`3bc75749-9349-407e-a00f-191a6db8450b`::moving from state preparing -> state finished >Thread-71930::DEBUG::2014-09-25 18:13:02,686::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71930::DEBUG::2014-09-25 18:13:02,686::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71930::DEBUG::2014-09-25 18:13:02,687::task::993::Storage.TaskManager.Task::(_decref) Task=`3bc75749-9349-407e-a00f-191a6db8450b`::ref 0 aborting False >Thread-71930::DEBUG::2014-09-25 18:13:02,689::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:02,704::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:02,705::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71931::DEBUG::2014-09-25 18:13:02,707::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:02,764::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:02,790::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144225 s, 71.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:13:04,810::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:04,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134182 s, 76.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:04,931::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:04,931::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:04,932::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:04,932::__init__::232::IOProcess::(_processLogs) (110794) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:04,933::__init__::232::IOProcess::(_processLogs) (110794) Queuing response >Thread-79::DEBUG::2014-09-25 18:13:04,934::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:13:04,942::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000482652 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:04,944::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:04,944::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:04,944::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:04,944::__init__::232::IOProcess::(_processLogs) (110795) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:04,945::__init__::232::IOProcess::(_processLogs) (110795) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:04,946::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:04,947::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:04,948::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:04,948::__init__::232::IOProcess::(_processLogs) (110796) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:04,948::__init__::232::IOProcess::(_processLogs) (110796) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:04,949::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:04,949::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:04,950::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:04,950::__init__::232::IOProcess::(_processLogs) (110797) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:04,950::__init__::232::IOProcess::(_processLogs) (110797) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:04,951::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:04,951::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:04,951::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:04,952::__init__::232::IOProcess::(_processLogs) (110798) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:04,953::__init__::232::IOProcess::(_processLogs) (110798) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:05,748::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:05,749::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71932::DEBUG::2014-09-25 18:13:05,751::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:06,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:06,884::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186245 s, 55.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:08,026::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:08,027::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:08,027::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:08,027::__init__::232::IOProcess::(_processLogs) (110799) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:13:08,028::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:08,028::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:08,028::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:08,028::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:08,029::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:08,029::__init__::232::IOProcess::(_processLogs) (110800) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:13:08,029::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:08,029::__init__::232::IOProcess::(_processLogs) (110801) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:13:08,030::__init__::232::IOProcess::(_processLogs) (110799) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:08,031::__init__::232::IOProcess::(_processLogs) (110801) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:08,603::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:08,603::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:08,604::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:08,604::__init__::232::IOProcess::(_processLogs) (110802) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:08,615::__init__::232::IOProcess::(_processLogs) (110802) Queuing response >Thread-81::DEBUG::2014-09-25 18:13:08,616::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:13:08,641::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0054798 s, 67.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:08,642::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:08,642::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:08,643::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:08,643::__init__::232::IOProcess::(_processLogs) (110803) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:08,646::__init__::232::IOProcess::(_processLogs) (110803) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:08,768::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:08,769::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71936::DEBUG::2014-09-25 18:13:08,771::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:08,904::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:08,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141871 s, 72.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:13:10,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:10,970::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138689 s, 73.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:11,493::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71937::DEBUG::2014-09-25 18:13:11,494::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:13:11,494::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71937::DEBUG::2014-09-25 18:13:11,497::task::595::Storage.TaskManager.Task::(_updateState) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::moving from state init -> state preparing >Thread-71937::INFO::2014-09-25 18:13:11,497::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71937::ERROR::2014-09-25 18:13:11,497::task::866::Storage.TaskManager.Task::(_setError) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71937::DEBUG::2014-09-25 18:13:11,498::task::885::Storage.TaskManager.Task::(_run) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::Task._run: 26a4cfd0-7055-4359-a254-6e22c9ead22a (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71937::DEBUG::2014-09-25 18:13:11,498::task::1217::Storage.TaskManager.Task::(stop) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::stopping in state preparing (force False) >Thread-71937::DEBUG::2014-09-25 18:13:11,498::task::993::Storage.TaskManager.Task::(_decref) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::ref 1 aborting True >Thread-71937::INFO::2014-09-25 18:13:11,499::task::1171::Storage.TaskManager.Task::(prepare) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::aborting: Task is aborted: u'' - code 100 >Thread-71937::DEBUG::2014-09-25 18:13:11,499::task::1176::Storage.TaskManager.Task::(prepare) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::Prepare: aborted: >Thread-71937::DEBUG::2014-09-25 18:13:11,499::task::993::Storage.TaskManager.Task::(_decref) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::ref 0 aborting True >Thread-71937::DEBUG::2014-09-25 18:13:11,499::task::928::Storage.TaskManager.Task::(_doAbort) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::Task._doAbort: force False >Thread-71937::DEBUG::2014-09-25 18:13:11,500::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71937::DEBUG::2014-09-25 18:13:11,500::task::595::Storage.TaskManager.Task::(_updateState) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::moving from state preparing -> state aborting >Thread-71937::DEBUG::2014-09-25 18:13:11,500::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::_aborting: recover policy none >Thread-71937::DEBUG::2014-09-25 18:13:11,500::task::595::Storage.TaskManager.Task::(_updateState) Task=`26a4cfd0-7055-4359-a254-6e22c9ead22a`::moving from state aborting -> state failed >Thread-71937::DEBUG::2014-09-25 18:13:11,500::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71937::DEBUG::2014-09-25 18:13:11,501::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71937::ERROR::2014-09-25 18:13:11,501::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71937::DEBUG::2014-09-25 18:13:11,501::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:11,788::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:11,789::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71938::DEBUG::2014-09-25 18:13:11,791::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:12,989::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:13,012::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135498 s, 75.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:14,804::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:14,805::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71939::DEBUG::2014-09-25 18:13:14,808::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:13:14,953::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:14,953::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:14,954::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:14,954::__init__::232::IOProcess::(_processLogs) (110804) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:14,965::__init__::232::IOProcess::(_processLogs) (110804) Queuing response >Thread-79::DEBUG::2014-09-25 18:13:14,966::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:13:14,985::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00242539 s, 232 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:14,986::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:14,986::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:14,986::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:14,987::__init__::232::IOProcess::(_processLogs) (110805) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:14,991::__init__::232::IOProcess::(_processLogs) (110805) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:14,992::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:14,993::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:14,994::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:14,994::__init__::232::IOProcess::(_processLogs) (110806) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:14,994::__init__::232::IOProcess::(_processLogs) (110806) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:14,995::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:14,995::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:14,996::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:14,996::__init__::232::IOProcess::(_processLogs) (110807) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:14,996::__init__::232::IOProcess::(_processLogs) (110807) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:14,997::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:14,997::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:14,998::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:14,999::__init__::232::IOProcess::(_processLogs) (110808) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:14,999::__init__::232::IOProcess::(_processLogs) (110808) Queuing response >Dummy-148::DEBUG::2014-09-25 18:13:15,031::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:15,071::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0243958 s, 42.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:13:17,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:17,154::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0236303 s, 43.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:17,820::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:17,821::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71940::DEBUG::2014-09-25 18:13:17,824::task::595::Storage.TaskManager.Task::(_updateState) Task=`511ba4a0-c6e8-4bd2-834e-859338d33392`::moving from state init -> state preparing >Thread-71940::INFO::2014-09-25 18:13:17,825::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71940::INFO::2014-09-25 18:13:17,825::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00242539', 'lastCheck': '2.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.8', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0054798', 'lastCheck': '9.2', 'valid': True}} >Thread-71940::DEBUG::2014-09-25 18:13:17,826::task::1191::Storage.TaskManager.Task::(prepare) Task=`511ba4a0-c6e8-4bd2-834e-859338d33392`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00242539', 'lastCheck': '2.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '19.8', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0054798', 'lastCheck': '9.2', 'valid': True}} >Thread-71940::DEBUG::2014-09-25 18:13:17,826::task::595::Storage.TaskManager.Task::(_updateState) Task=`511ba4a0-c6e8-4bd2-834e-859338d33392`::moving from state preparing -> state finished >Thread-71940::DEBUG::2014-09-25 18:13:17,826::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71940::DEBUG::2014-09-25 18:13:17,826::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71940::DEBUG::2014-09-25 18:13:17,826::task::993::Storage.TaskManager.Task::(_decref) Task=`511ba4a0-c6e8-4bd2-834e-859338d33392`::ref 0 aborting False >Thread-71940::DEBUG::2014-09-25 18:13:17,829::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:17,852::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:17,853::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71941::DEBUG::2014-09-25 18:13:17,855::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:13:18,648::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:18,648::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:18,649::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:18,649::__init__::232::IOProcess::(_processLogs) (110809) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:18,655::__init__::232::IOProcess::(_processLogs) (110809) Queuing response >Thread-81::DEBUG::2014-09-25 18:13:18,656::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:13:18,688::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00518079 s, 71.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:18,689::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:18,689::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:18,689::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:18,690::__init__::232::IOProcess::(_processLogs) (110810) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:18,693::__init__::232::IOProcess::(_processLogs) (110810) Queuing response >Dummy-148::DEBUG::2014-09-25 18:13:19,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:19,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0313962 s, 32.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:20,909::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:20,910::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71942::DEBUG::2014-09-25 18:13:20,912::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:21,234::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:21,267::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171402 s, 59.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:21,527::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71943::DEBUG::2014-09-25 18:13:21,528::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:13:21,528::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71943::DEBUG::2014-09-25 18:13:21,531::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::moving from state init -> state preparing >Thread-71943::INFO::2014-09-25 18:13:21,531::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71943::ERROR::2014-09-25 18:13:21,532::task::866::Storage.TaskManager.Task::(_setError) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71943::DEBUG::2014-09-25 18:13:21,532::task::885::Storage.TaskManager.Task::(_run) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::Task._run: 8ee3720e-b617-4a28-a561-0eabf842134e (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71943::DEBUG::2014-09-25 18:13:21,532::task::1217::Storage.TaskManager.Task::(stop) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::stopping in state preparing (force False) >Thread-71943::DEBUG::2014-09-25 18:13:21,533::task::993::Storage.TaskManager.Task::(_decref) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::ref 1 aborting True >Thread-71943::INFO::2014-09-25 18:13:21,533::task::1171::Storage.TaskManager.Task::(prepare) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::aborting: Task is aborted: u'' - code 100 >Thread-71943::DEBUG::2014-09-25 18:13:21,533::task::1176::Storage.TaskManager.Task::(prepare) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::Prepare: aborted: >Thread-71943::DEBUG::2014-09-25 18:13:21,533::task::993::Storage.TaskManager.Task::(_decref) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::ref 0 aborting True >Thread-71943::DEBUG::2014-09-25 18:13:21,534::task::928::Storage.TaskManager.Task::(_doAbort) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::Task._doAbort: force False >Thread-71943::DEBUG::2014-09-25 18:13:21,534::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71943::DEBUG::2014-09-25 18:13:21,534::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::moving from state preparing -> state aborting >Thread-71943::DEBUG::2014-09-25 18:13:21,534::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::_aborting: recover policy none >Thread-71943::DEBUG::2014-09-25 18:13:21,534::task::595::Storage.TaskManager.Task::(_updateState) Task=`8ee3720e-b617-4a28-a561-0eabf842134e`::moving from state aborting -> state failed >Thread-71943::DEBUG::2014-09-25 18:13:21,535::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71943::DEBUG::2014-09-25 18:13:21,535::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71943::ERROR::2014-09-25 18:13:21,535::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71943::DEBUG::2014-09-25 18:13:21,536::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:23,287::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:23,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164134 s, 62.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:23,930::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:23,932::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71944::DEBUG::2014-09-25 18:13:23,934::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:13:24,999::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:25,000::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:25,000::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:25,000::__init__::232::IOProcess::(_processLogs) (110811) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:25,015::__init__::232::IOProcess::(_processLogs) (110811) Queuing response >Thread-79::DEBUG::2014-09-25 18:13:25,017::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:13:25,035::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00648329 s, 86.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:25,036::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:25,037::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:25,037::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:25,037::__init__::232::IOProcess::(_processLogs) (110812) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:25,040::__init__::232::IOProcess::(_processLogs) (110812) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:25,042::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:25,042::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:25,042::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:25,043::__init__::232::IOProcess::(_processLogs) (110813) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:25,043::__init__::232::IOProcess::(_processLogs) (110813) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:25,044::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:25,044::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:25,045::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:25,045::__init__::232::IOProcess::(_processLogs) (110814) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:25,045::__init__::232::IOProcess::(_processLogs) (110814) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:25,046::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:25,046::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:25,047::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:25,048::__init__::232::IOProcess::(_processLogs) (110815) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:25,048::__init__::232::IOProcess::(_processLogs) (110815) Queuing response >Dummy-148::DEBUG::2014-09-25 18:13:25,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:25,359::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0139506 s, 73.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:26,954::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:26,955::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71945::DEBUG::2014-09-25 18:13:26,957::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:27,378::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:27,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166918 s, 61.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:28,695::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:28,696::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:28,696::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:28,696::__init__::232::IOProcess::(_processLogs) (110816) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:28,697::__init__::232::IOProcess::(_processLogs) (110816) Queuing response >Thread-81::DEBUG::2014-09-25 18:13:28,698::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:13:28,707::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000341821 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:28,708::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:28,708::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:28,709::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:28,710::__init__::232::IOProcess::(_processLogs) (110817) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:28,711::__init__::232::IOProcess::(_processLogs) (110817) Queuing response >Dummy-148::DEBUG::2014-09-25 18:13:29,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:29,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172277 s, 59.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:29,972::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:29,974::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71946::DEBUG::2014-09-25 18:13:29,976::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:31,473::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:31,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132947 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:31,570::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:31,572::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71947::DEBUG::2014-09-25 18:13:31,572::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71947::DEBUG::2014-09-25 18:13:31,574::task::595::Storage.TaskManager.Task::(_updateState) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::moving from state init -> state preparing >Thread-71947::INFO::2014-09-25 18:13:31,574::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71947::ERROR::2014-09-25 18:13:31,574::task::866::Storage.TaskManager.Task::(_setError) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71947::DEBUG::2014-09-25 18:13:31,575::task::885::Storage.TaskManager.Task::(_run) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::Task._run: 2306deff-a02d-4aca-9fc4-c805cc85dba6 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71947::DEBUG::2014-09-25 18:13:31,575::task::1217::Storage.TaskManager.Task::(stop) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::stopping in state preparing (force False) >Thread-71947::DEBUG::2014-09-25 18:13:31,575::task::993::Storage.TaskManager.Task::(_decref) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::ref 1 aborting True >Thread-71947::INFO::2014-09-25 18:13:31,576::task::1171::Storage.TaskManager.Task::(prepare) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::aborting: Task is aborted: u'' - code 100 >Thread-71947::DEBUG::2014-09-25 18:13:31,576::task::1176::Storage.TaskManager.Task::(prepare) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::Prepare: aborted: >Thread-71947::DEBUG::2014-09-25 18:13:31,576::task::993::Storage.TaskManager.Task::(_decref) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::ref 0 aborting True >Thread-71947::DEBUG::2014-09-25 18:13:31,576::task::928::Storage.TaskManager.Task::(_doAbort) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::Task._doAbort: force False >Thread-71947::DEBUG::2014-09-25 18:13:31,577::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71947::DEBUG::2014-09-25 18:13:31,577::task::595::Storage.TaskManager.Task::(_updateState) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::moving from state preparing -> state aborting >Thread-71947::DEBUG::2014-09-25 18:13:31,577::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::_aborting: recover policy none >Thread-71947::DEBUG::2014-09-25 18:13:31,577::task::595::Storage.TaskManager.Task::(_updateState) Task=`2306deff-a02d-4aca-9fc4-c805cc85dba6`::moving from state aborting -> state failed >Thread-71947::DEBUG::2014-09-25 18:13:31,577::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71947::DEBUG::2014-09-25 18:13:31,578::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71947::ERROR::2014-09-25 18:13:31,578::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71947::DEBUG::2014-09-25 18:13:31,578::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:32,991::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:32,992::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71948::DEBUG::2014-09-25 18:13:32,994::task::595::Storage.TaskManager.Task::(_updateState) Task=`818755fa-09ad-4629-b5e3-591787b4ed1c`::moving from state init -> state preparing >Thread-71948::INFO::2014-09-25 18:13:32,995::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71948::INFO::2014-09-25 18:13:32,995::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00648329', 'lastCheck': '7.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '35.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000341821', 'lastCheck': '4.3', 'valid': True}} >Thread-71948::DEBUG::2014-09-25 18:13:32,995::task::1191::Storage.TaskManager.Task::(prepare) Task=`818755fa-09ad-4629-b5e3-591787b4ed1c`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00648329', 'lastCheck': '7.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '35.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000341821', 'lastCheck': '4.3', 'valid': True}} >Thread-71948::DEBUG::2014-09-25 18:13:32,996::task::595::Storage.TaskManager.Task::(_updateState) Task=`818755fa-09ad-4629-b5e3-591787b4ed1c`::moving from state preparing -> state finished >Thread-71948::DEBUG::2014-09-25 18:13:32,996::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71948::DEBUG::2014-09-25 18:13:32,996::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71948::DEBUG::2014-09-25 18:13:32,996::task::993::Storage.TaskManager.Task::(_decref) Task=`818755fa-09ad-4629-b5e3-591787b4ed1c`::ref 0 aborting False >Thread-71948::DEBUG::2014-09-25 18:13:32,999::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:33,025::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:33,026::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71949::DEBUG::2014-09-25 18:13:33,027::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:33,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:33,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0187884 s, 54.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:35,048::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:35,049::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:35,049::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:35,049::__init__::232::IOProcess::(_processLogs) (110818) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:35,050::__init__::232::IOProcess::(_processLogs) (110818) Queuing response >Thread-79::DEBUG::2014-09-25 18:13:35,052::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:13:35,068::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00699698 s, 80.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:35,069::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:35,070::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:35,070::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:35,070::__init__::232::IOProcess::(_processLogs) (110819) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:35,071::__init__::232::IOProcess::(_processLogs) (110819) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:35,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:35,073::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:35,074::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:35,074::__init__::232::IOProcess::(_processLogs) (110820) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:35,074::__init__::232::IOProcess::(_processLogs) (110820) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:35,075::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:35,075::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:35,076::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:35,076::__init__::232::IOProcess::(_processLogs) (110821) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:35,076::__init__::232::IOProcess::(_processLogs) (110821) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:35,077::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:35,077::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:35,078::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:35,079::__init__::232::IOProcess::(_processLogs) (110822) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:35,079::__init__::232::IOProcess::(_processLogs) (110822) Queuing response >Dummy-148::DEBUG::2014-09-25 18:13:35,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:35,591::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136291 s, 75.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:36,105::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:36,107::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71950::DEBUG::2014-09-25 18:13:36,109::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:37,611::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:37,641::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173947 s, 58.9 MB/s\n'; <rc> = 0 >Thread-71917::WARNING::2014-09-25 18:13:38,476::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-29::DEBUG::2014-09-25 18:13:38,711::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:38,711::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:38,712::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:38,712::__init__::232::IOProcess::(_processLogs) (110823) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:38,722::__init__::232::IOProcess::(_processLogs) (110823) Queuing response >Thread-81::DEBUG::2014-09-25 18:13:38,723::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:13:38,770::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0186813 s, 19.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:38,772::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:38,772::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:38,772::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:38,772::__init__::232::IOProcess::(_processLogs) (110824) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:38,779::__init__::232::IOProcess::(_processLogs) (110824) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:39,128::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:39,129::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71951::DEBUG::2014-09-25 18:13:39,131::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:39,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:39,693::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167914 s, 61.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:41,605::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71952::DEBUG::2014-09-25 18:13:41,606::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:13:41,606::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71952::DEBUG::2014-09-25 18:13:41,608::task::595::Storage.TaskManager.Task::(_updateState) Task=`648361be-8386-471e-9f29-20b2bd713204`::moving from state init -> state preparing >Thread-71952::INFO::2014-09-25 18:13:41,609::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71952::ERROR::2014-09-25 18:13:41,609::task::866::Storage.TaskManager.Task::(_setError) Task=`648361be-8386-471e-9f29-20b2bd713204`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71952::DEBUG::2014-09-25 18:13:41,610::task::885::Storage.TaskManager.Task::(_run) Task=`648361be-8386-471e-9f29-20b2bd713204`::Task._run: 648361be-8386-471e-9f29-20b2bd713204 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71952::DEBUG::2014-09-25 18:13:41,610::task::1217::Storage.TaskManager.Task::(stop) Task=`648361be-8386-471e-9f29-20b2bd713204`::stopping in state preparing (force False) >Thread-71952::DEBUG::2014-09-25 18:13:41,610::task::993::Storage.TaskManager.Task::(_decref) Task=`648361be-8386-471e-9f29-20b2bd713204`::ref 1 aborting True >Thread-71952::INFO::2014-09-25 18:13:41,610::task::1171::Storage.TaskManager.Task::(prepare) Task=`648361be-8386-471e-9f29-20b2bd713204`::aborting: Task is aborted: u'' - code 100 >Thread-71952::DEBUG::2014-09-25 18:13:41,610::task::1176::Storage.TaskManager.Task::(prepare) Task=`648361be-8386-471e-9f29-20b2bd713204`::Prepare: aborted: >Thread-71952::DEBUG::2014-09-25 18:13:41,611::task::993::Storage.TaskManager.Task::(_decref) Task=`648361be-8386-471e-9f29-20b2bd713204`::ref 0 aborting True >Thread-71952::DEBUG::2014-09-25 18:13:41,611::task::928::Storage.TaskManager.Task::(_doAbort) Task=`648361be-8386-471e-9f29-20b2bd713204`::Task._doAbort: force False >Thread-71952::DEBUG::2014-09-25 18:13:41,611::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71952::DEBUG::2014-09-25 18:13:41,611::task::595::Storage.TaskManager.Task::(_updateState) Task=`648361be-8386-471e-9f29-20b2bd713204`::moving from state preparing -> state aborting >Thread-71952::DEBUG::2014-09-25 18:13:41,612::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`648361be-8386-471e-9f29-20b2bd713204`::_aborting: recover policy none >Thread-71952::DEBUG::2014-09-25 18:13:41,612::task::595::Storage.TaskManager.Task::(_updateState) Task=`648361be-8386-471e-9f29-20b2bd713204`::moving from state aborting -> state failed >Thread-71952::DEBUG::2014-09-25 18:13:41,612::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71952::DEBUG::2014-09-25 18:13:41,612::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71952::ERROR::2014-09-25 18:13:41,613::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71952::DEBUG::2014-09-25 18:13:41,613::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:41,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:41,739::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146782 s, 69.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:42,146::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:42,147::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71953::DEBUG::2014-09-25 18:13:42,149::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:43,758::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:43,984::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.211948 s, 4.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:45,079::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:45,080::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:45,080::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:45,080::__init__::232::IOProcess::(_processLogs) (110825) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:45,083::__init__::232::IOProcess::(_processLogs) (110825) Queuing response >Thread-79::DEBUG::2014-09-25 18:13:45,084::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:13:45,096::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00246185 s, 228 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:45,097::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:45,097::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:45,098::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:45,098::__init__::232::IOProcess::(_processLogs) (110826) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:45,099::__init__::232::IOProcess::(_processLogs) (110826) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:45,100::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:45,101::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:45,102::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:45,102::__init__::232::IOProcess::(_processLogs) (110827) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:45,102::__init__::232::IOProcess::(_processLogs) (110827) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:45,103::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:45,103::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:45,104::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:45,104::__init__::232::IOProcess::(_processLogs) (110828) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:45,104::__init__::232::IOProcess::(_processLogs) (110828) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:45,105::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:45,105::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:45,106::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:45,107::__init__::232::IOProcess::(_processLogs) (110829) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:45,107::__init__::232::IOProcess::(_processLogs) (110829) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:45,161::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:45,163::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71954::DEBUG::2014-09-25 18:13:45,165::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:46,004::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:46,038::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.023468 s, 43.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:48,179::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:48,180::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71955::DEBUG::2014-09-25 18:13:48,183::task::595::Storage.TaskManager.Task::(_updateState) Task=`d237d41d-55a7-47b3-a656-e43a1e4edeea`::moving from state init -> state preparing >Thread-71955::INFO::2014-09-25 18:13:48,183::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71955::INFO::2014-09-25 18:13:48,184::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00246185', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '50.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0186813', 'lastCheck': '9.4', 'valid': True}} >Thread-71955::DEBUG::2014-09-25 18:13:48,184::task::1191::Storage.TaskManager.Task::(prepare) Task=`d237d41d-55a7-47b3-a656-e43a1e4edeea`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00246185', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '50.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0186813', 'lastCheck': '9.4', 'valid': True}} >Thread-71955::DEBUG::2014-09-25 18:13:48,184::task::595::Storage.TaskManager.Task::(_updateState) Task=`d237d41d-55a7-47b3-a656-e43a1e4edeea`::moving from state preparing -> state finished >Thread-71955::DEBUG::2014-09-25 18:13:48,184::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71955::DEBUG::2014-09-25 18:13:48,185::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71955::DEBUG::2014-09-25 18:13:48,185::task::993::Storage.TaskManager.Task::(_decref) Task=`d237d41d-55a7-47b3-a656-e43a1e4edeea`::ref 0 aborting False >Thread-71955::DEBUG::2014-09-25 18:13:48,187::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:48,202::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:48,203::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71956::DEBUG::2014-09-25 18:13:48,205::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:48,250::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:48,277::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156166 s, 65.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:48,781::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:48,781::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:48,782::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:48,783::__init__::232::IOProcess::(_processLogs) (110830) Got request for method 'statvfs' >Thread-81::DEBUG::2014-09-25 18:13:48,783::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:13:48,784::__init__::232::IOProcess::(_processLogs) (110830) Queuing response >Thread-81::DEBUG::2014-09-25 18:13:48,793::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000396106 s, 937 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:48,794::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:48,795::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:48,795::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:48,795::__init__::232::IOProcess::(_processLogs) (110831) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:48,796::__init__::232::IOProcess::(_processLogs) (110831) Queuing response >Dummy-148::DEBUG::2014-09-25 18:13:50,297::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:50,333::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215634 s, 47.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:51,258::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:51,259::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71957::DEBUG::2014-09-25 18:13:51,261::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:51,637::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:51,638::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71958::DEBUG::2014-09-25 18:13:51,638::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71958::DEBUG::2014-09-25 18:13:51,640::task::595::Storage.TaskManager.Task::(_updateState) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::moving from state init -> state preparing >Thread-71958::INFO::2014-09-25 18:13:51,640::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71958::ERROR::2014-09-25 18:13:51,640::task::866::Storage.TaskManager.Task::(_setError) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71958::DEBUG::2014-09-25 18:13:51,641::task::885::Storage.TaskManager.Task::(_run) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::Task._run: fbccfbc9-ded8-467a-8732-5a80b5777af4 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71958::DEBUG::2014-09-25 18:13:51,641::task::1217::Storage.TaskManager.Task::(stop) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::stopping in state preparing (force False) >Thread-71958::DEBUG::2014-09-25 18:13:51,641::task::993::Storage.TaskManager.Task::(_decref) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::ref 1 aborting True >Thread-71958::INFO::2014-09-25 18:13:51,642::task::1171::Storage.TaskManager.Task::(prepare) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::aborting: Task is aborted: u'' - code 100 >Thread-71958::DEBUG::2014-09-25 18:13:51,642::task::1176::Storage.TaskManager.Task::(prepare) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::Prepare: aborted: >Thread-71958::DEBUG::2014-09-25 18:13:51,642::task::993::Storage.TaskManager.Task::(_decref) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::ref 0 aborting True >Thread-71958::DEBUG::2014-09-25 18:13:51,642::task::928::Storage.TaskManager.Task::(_doAbort) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::Task._doAbort: force False >Thread-71958::DEBUG::2014-09-25 18:13:51,642::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71958::DEBUG::2014-09-25 18:13:51,643::task::595::Storage.TaskManager.Task::(_updateState) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::moving from state preparing -> state aborting >Thread-71958::DEBUG::2014-09-25 18:13:51,643::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::_aborting: recover policy none >Thread-71958::DEBUG::2014-09-25 18:13:51,643::task::595::Storage.TaskManager.Task::(_updateState) Task=`fbccfbc9-ded8-467a-8732-5a80b5777af4`::moving from state aborting -> state failed >Thread-71958::DEBUG::2014-09-25 18:13:51,643::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71958::DEBUG::2014-09-25 18:13:51,644::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71958::ERROR::2014-09-25 18:13:51,644::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71958::DEBUG::2014-09-25 18:13:51,644::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:52,352::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:52,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0187712 s, 54.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:54,278::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:54,279::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71959::DEBUG::2014-09-25 18:13:54,281::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:54,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:54,430::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016637 s, 61.5 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 18:13:55,106::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 18:13:55,111::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,112::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,112::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,112::__init__::232::IOProcess::(_processLogs) (110832) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:13:55,113::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,113::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,113::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,113::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,114::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,114::__init__::232::IOProcess::(_processLogs) (110833) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:13:55,114::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,114::__init__::232::IOProcess::(_processLogs) (110834) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:13:55,187::__init__::232::IOProcess::(_processLogs) (110832) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:55,188::__init__::232::IOProcess::(_processLogs) (110834) Queuing response >Thread-79::DEBUG::2014-09-25 18:13:55,192::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 18:13:55,193::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,194::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,194::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,194::__init__::232::IOProcess::(_processLogs) (110835) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 18:13:55,234::__init__::232::IOProcess::(_processLogs) (110835) Queuing response >Thread-79::DEBUG::2014-09-25 18:13:55,235::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:13:55,235::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,236::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,236::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,236::__init__::232::IOProcess::(_processLogs) (110836) Got request for method 'readfile' >Thread-29::DEBUG::2014-09-25 18:13:55,251::__init__::232::IOProcess::(_processLogs) (110836) Queuing response >Thread-79::DEBUG::2014-09-25 18:13:55,252::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 18:13:55,253::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,254::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,254::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,255::__init__::232::IOProcess::(_processLogs) (110837) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:55,255::__init__::232::IOProcess::(_processLogs) (110837) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:55,255::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,256::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,256::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,256::__init__::232::IOProcess::(_processLogs) (110838) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:13:55,263::__init__::232::IOProcess::(_processLogs) (110838) Queuing response >Thread-79::DEBUG::2014-09-25 18:13:55,264::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-79::INFO::2014-09-25 18:13:55,264::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 18:13:55,264::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 18:13:55,265::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,265::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,265::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,265::__init__::232::IOProcess::(_processLogs) (110839) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:55,269::__init__::232::IOProcess::(_processLogs) (110839) Queuing response >Thread-79::DEBUG::2014-09-25 18:13:55,269::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:13:55,287::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00624301 s, 90.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:55,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,288::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,289::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,289::__init__::232::IOProcess::(_processLogs) (110840) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:55,292::__init__::232::IOProcess::(_processLogs) (110840) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:55,294::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,294::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,294::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,295::__init__::232::IOProcess::(_processLogs) (110841) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:55,297::__init__::232::IOProcess::(_processLogs) (110841) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:55,297::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,298::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,298::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,298::__init__::232::IOProcess::(_processLogs) (110842) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:55,300::__init__::232::IOProcess::(_processLogs) (110842) Queuing response >Thread-29::DEBUG::2014-09-25 18:13:55,301::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:55,301::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:55,301::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:55,302::__init__::232::IOProcess::(_processLogs) (110843) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:13:55,306::__init__::232::IOProcess::(_processLogs) (110843) Queuing response >Dummy-148::DEBUG::2014-09-25 18:13:56,449::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:56,471::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0131714 s, 77.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:13:57,303::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:13:57,305::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71963::DEBUG::2014-09-25 18:13:57,307::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:13:58,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:13:58,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0160931 s, 63.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:58,798::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:58,799::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:58,799::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:58,799::__init__::232::IOProcess::(_processLogs) (110844) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:58,805::__init__::232::IOProcess::(_processLogs) (110844) Queuing response >Thread-81::DEBUG::2014-09-25 18:13:58,806::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:13:58,824::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0042931 s, 86.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:13:58,825::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:13:58,825::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:13:58,826::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:13:58,826::__init__::232::IOProcess::(_processLogs) (110845) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:13:58,828::__init__::232::IOProcess::(_processLogs) (110845) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:00,321::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:00,322::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71964::DEBUG::2014-09-25 18:14:00,324::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:00,540::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:00,569::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166977 s, 61.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:01,678::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71965::DEBUG::2014-09-25 18:14:01,679::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:14:01,679::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71965::DEBUG::2014-09-25 18:14:01,681::task::595::Storage.TaskManager.Task::(_updateState) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::moving from state init -> state preparing >Thread-71965::INFO::2014-09-25 18:14:01,682::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71965::ERROR::2014-09-25 18:14:01,682::task::866::Storage.TaskManager.Task::(_setError) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71965::DEBUG::2014-09-25 18:14:01,683::task::885::Storage.TaskManager.Task::(_run) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::Task._run: d121bb7d-33b4-40f7-8b05-d827e4ed21ab (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71965::DEBUG::2014-09-25 18:14:01,683::task::1217::Storage.TaskManager.Task::(stop) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::stopping in state preparing (force False) >Thread-71965::DEBUG::2014-09-25 18:14:01,683::task::993::Storage.TaskManager.Task::(_decref) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::ref 1 aborting True >Thread-71965::INFO::2014-09-25 18:14:01,683::task::1171::Storage.TaskManager.Task::(prepare) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::aborting: Task is aborted: u'' - code 100 >Thread-71965::DEBUG::2014-09-25 18:14:01,684::task::1176::Storage.TaskManager.Task::(prepare) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::Prepare: aborted: >Thread-71965::DEBUG::2014-09-25 18:14:01,684::task::993::Storage.TaskManager.Task::(_decref) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::ref 0 aborting True >Thread-71965::DEBUG::2014-09-25 18:14:01,684::task::928::Storage.TaskManager.Task::(_doAbort) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::Task._doAbort: force False >Thread-71965::DEBUG::2014-09-25 18:14:01,684::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71965::DEBUG::2014-09-25 18:14:01,684::task::595::Storage.TaskManager.Task::(_updateState) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::moving from state preparing -> state aborting >Thread-71965::DEBUG::2014-09-25 18:14:01,685::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::_aborting: recover policy none >Thread-71965::DEBUG::2014-09-25 18:14:01,685::task::595::Storage.TaskManager.Task::(_updateState) Task=`d121bb7d-33b4-40f7-8b05-d827e4ed21ab`::moving from state aborting -> state failed >Thread-71965::DEBUG::2014-09-25 18:14:01,685::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71965::DEBUG::2014-09-25 18:14:01,685::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71965::ERROR::2014-09-25 18:14:01,686::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71965::DEBUG::2014-09-25 18:14:01,686::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:02,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:02,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148735 s, 68.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:03,336::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:03,338::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71966::DEBUG::2014-09-25 18:14:03,340::task::595::Storage.TaskManager.Task::(_updateState) Task=`6490e0c0-c308-47fe-b719-2ae0787cd29d`::moving from state init -> state preparing >Thread-71966::INFO::2014-09-25 18:14:03,341::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71966::INFO::2014-09-25 18:14:03,341::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00624301', 'lastCheck': '8.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '65.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0042931', 'lastCheck': '4.5', 'valid': True}} >Thread-71966::DEBUG::2014-09-25 18:14:03,341::task::1191::Storage.TaskManager.Task::(prepare) Task=`6490e0c0-c308-47fe-b719-2ae0787cd29d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00624301', 'lastCheck': '8.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '65.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0042931', 'lastCheck': '4.5', 'valid': True}} >Thread-71966::DEBUG::2014-09-25 18:14:03,342::task::595::Storage.TaskManager.Task::(_updateState) Task=`6490e0c0-c308-47fe-b719-2ae0787cd29d`::moving from state preparing -> state finished >Thread-71966::DEBUG::2014-09-25 18:14:03,342::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71966::DEBUG::2014-09-25 18:14:03,342::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71966::DEBUG::2014-09-25 18:14:03,342::task::993::Storage.TaskManager.Task::(_decref) Task=`6490e0c0-c308-47fe-b719-2ae0787cd29d`::ref 0 aborting False >Thread-71966::DEBUG::2014-09-25 18:14:03,345::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:03,372::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:03,373::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71967::DEBUG::2014-09-25 18:14:03,375::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:04,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:04,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168945 s, 60.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:05,307::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:05,307::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:05,308::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:05,308::__init__::232::IOProcess::(_processLogs) (110846) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:05,309::__init__::232::IOProcess::(_processLogs) (110846) Queuing response >Thread-79::DEBUG::2014-09-25 18:14:05,310::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:14:05,347::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00878283 s, 64.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:05,348::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:05,348::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:05,348::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:05,349::__init__::232::IOProcess::(_processLogs) (110847) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:05,355::__init__::232::IOProcess::(_processLogs) (110847) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:05,357::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:05,357::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:05,358::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:05,358::__init__::232::IOProcess::(_processLogs) (110848) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:05,358::__init__::232::IOProcess::(_processLogs) (110848) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:05,359::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:05,359::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:05,360::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:05,360::__init__::232::IOProcess::(_processLogs) (110849) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:05,361::__init__::232::IOProcess::(_processLogs) (110849) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:05,361::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:05,361::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:05,363::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:05,363::__init__::232::IOProcess::(_processLogs) (110850) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:05,363::__init__::232::IOProcess::(_processLogs) (110850) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:06,422::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:06,424::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71968::DEBUG::2014-09-25 18:14:06,427::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:06,680::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:06,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0141772 s, 72.2 MB/s\n'; <rc> = 0 >Thread-71934::WARNING::2014-09-25 18:14:08,025::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:14:08,026::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:14:08,027::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Dummy-148::DEBUG::2014-09-25 18:14:08,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:08,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214357 s, 47.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:08,830::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:08,830::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:08,830::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:08,830::__init__::232::IOProcess::(_processLogs) (110851) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:08,837::__init__::232::IOProcess::(_processLogs) (110851) Queuing response >Thread-81::DEBUG::2014-09-25 18:14:08,838::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:14:08,858::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00174369 s, 213 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:08,859::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:08,859::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:08,860::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:08,860::__init__::232::IOProcess::(_processLogs) (110852) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:08,862::__init__::232::IOProcess::(_processLogs) (110852) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:09,440::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:09,442::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71969::DEBUG::2014-09-25 18:14:09,444::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:10,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:10,813::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203521 s, 50.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:11,716::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71970::DEBUG::2014-09-25 18:14:11,717::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:14:11,717::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71970::DEBUG::2014-09-25 18:14:11,719::task::595::Storage.TaskManager.Task::(_updateState) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::moving from state init -> state preparing >Thread-71970::INFO::2014-09-25 18:14:11,720::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71970::ERROR::2014-09-25 18:14:11,720::task::866::Storage.TaskManager.Task::(_setError) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71970::DEBUG::2014-09-25 18:14:11,721::task::885::Storage.TaskManager.Task::(_run) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::Task._run: f2b2340f-29c5-40c0-8ae3-9f6445454964 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71970::DEBUG::2014-09-25 18:14:11,721::task::1217::Storage.TaskManager.Task::(stop) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::stopping in state preparing (force False) >Thread-71970::DEBUG::2014-09-25 18:14:11,721::task::993::Storage.TaskManager.Task::(_decref) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::ref 1 aborting True >Thread-71970::INFO::2014-09-25 18:14:11,721::task::1171::Storage.TaskManager.Task::(prepare) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::aborting: Task is aborted: u'' - code 100 >Thread-71970::DEBUG::2014-09-25 18:14:11,722::task::1176::Storage.TaskManager.Task::(prepare) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::Prepare: aborted: >Thread-71970::DEBUG::2014-09-25 18:14:11,722::task::993::Storage.TaskManager.Task::(_decref) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::ref 0 aborting True >Thread-71970::DEBUG::2014-09-25 18:14:11,722::task::928::Storage.TaskManager.Task::(_doAbort) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::Task._doAbort: force False >Thread-71970::DEBUG::2014-09-25 18:14:11,722::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71970::DEBUG::2014-09-25 18:14:11,723::task::595::Storage.TaskManager.Task::(_updateState) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::moving from state preparing -> state aborting >Thread-71970::DEBUG::2014-09-25 18:14:11,723::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::_aborting: recover policy none >Thread-71970::DEBUG::2014-09-25 18:14:11,723::task::595::Storage.TaskManager.Task::(_updateState) Task=`f2b2340f-29c5-40c0-8ae3-9f6445454964`::moving from state aborting -> state failed >Thread-71970::DEBUG::2014-09-25 18:14:11,723::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71970::DEBUG::2014-09-25 18:14:11,723::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71970::ERROR::2014-09-25 18:14:11,724::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71970::DEBUG::2014-09-25 18:14:11,724::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:12,457::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:12,459::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71971::DEBUG::2014-09-25 18:14:12,460::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:12,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:12,855::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132846 s, 77.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:14:14,875::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:14,898::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136038 s, 75.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:15,364::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:15,364::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:15,364::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:15,364::__init__::232::IOProcess::(_processLogs) (110853) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:15,368::__init__::232::IOProcess::(_processLogs) (110853) Queuing response >Thread-79::DEBUG::2014-09-25 18:14:15,369::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:14:15,385::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00330697 s, 170 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:15,386::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:15,387::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:15,387::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:15,387::__init__::232::IOProcess::(_processLogs) (110854) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:15,392::__init__::232::IOProcess::(_processLogs) (110854) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:15,394::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:15,395::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:15,395::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:15,396::__init__::232::IOProcess::(_processLogs) (110855) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:15,396::__init__::232::IOProcess::(_processLogs) (110855) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:15,396::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:15,397::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:15,397::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:15,397::__init__::232::IOProcess::(_processLogs) (110856) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:15,397::__init__::232::IOProcess::(_processLogs) (110856) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:15,398::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:15,399::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:15,399::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:15,399::__init__::232::IOProcess::(_processLogs) (110857) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:15,399::__init__::232::IOProcess::(_processLogs) (110857) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:15,478::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:15,479::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71972::DEBUG::2014-09-25 18:14:15,481::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:16,918::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:16,957::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.025121 s, 40.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:18,032::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:18,032::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:18,032::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:18,032::__init__::232::IOProcess::(_processLogs) (110858) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:14:18,034::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:18,035::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:18,035::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:18,035::__init__::232::IOProcess::(_processLogs) (110859) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:14:18,036::__init__::232::IOProcess::(_processLogs) (110858) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:18,037::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:18,037::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:18,037::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:18,037::__init__::232::IOProcess::(_processLogs) (110860) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:14:18,039::__init__::232::IOProcess::(_processLogs) (110860) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:18,495::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:18,496::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71976::DEBUG::2014-09-25 18:14:18,498::task::595::Storage.TaskManager.Task::(_updateState) Task=`c8231dc8-962a-43e1-a8c7-f14fd1c87de7`::moving from state init -> state preparing >Thread-71976::INFO::2014-09-25 18:14:18,499::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71976::INFO::2014-09-25 18:14:18,499::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00330697', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00174369', 'lastCheck': '9.6', 'valid': True}} >Thread-71976::DEBUG::2014-09-25 18:14:18,499::task::1191::Storage.TaskManager.Task::(prepare) Task=`c8231dc8-962a-43e1-a8c7-f14fd1c87de7`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00330697', 'lastCheck': '3.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '10.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00174369', 'lastCheck': '9.6', 'valid': True}} >Thread-71976::DEBUG::2014-09-25 18:14:18,500::task::595::Storage.TaskManager.Task::(_updateState) Task=`c8231dc8-962a-43e1-a8c7-f14fd1c87de7`::moving from state preparing -> state finished >Thread-71976::DEBUG::2014-09-25 18:14:18,500::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71976::DEBUG::2014-09-25 18:14:18,500::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71976::DEBUG::2014-09-25 18:14:18,500::task::993::Storage.TaskManager.Task::(_decref) Task=`c8231dc8-962a-43e1-a8c7-f14fd1c87de7`::ref 0 aborting False >Thread-71976::DEBUG::2014-09-25 18:14:18,503::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:18,519::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:18,520::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71977::DEBUG::2014-09-25 18:14:18,521::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:14:18,864::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:18,864::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:18,864::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:18,865::__init__::232::IOProcess::(_processLogs) (110861) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:18,867::__init__::232::IOProcess::(_processLogs) (110861) Queuing response >Thread-81::DEBUG::2014-09-25 18:14:18,868::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:14:18,884::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00479981 s, 77.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:18,885::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:18,885::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:18,885::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:18,886::__init__::232::IOProcess::(_processLogs) (110862) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:18,889::__init__::232::IOProcess::(_processLogs) (110862) Queuing response >Dummy-148::DEBUG::2014-09-25 18:14:19,028::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:19,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159934 s, 64.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:14:21,074::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:21,096::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135806 s, 75.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:21,563::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:21,564::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71978::DEBUG::2014-09-25 18:14:21,566::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:21,751::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:21,752::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71979::DEBUG::2014-09-25 18:14:21,752::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71979::DEBUG::2014-09-25 18:14:21,754::task::595::Storage.TaskManager.Task::(_updateState) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::moving from state init -> state preparing >Thread-71979::INFO::2014-09-25 18:14:21,755::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71979::ERROR::2014-09-25 18:14:21,755::task::866::Storage.TaskManager.Task::(_setError) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71979::DEBUG::2014-09-25 18:14:21,756::task::885::Storage.TaskManager.Task::(_run) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::Task._run: 279ac599-e3b1-4e32-ba68-5a129638228f (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71979::DEBUG::2014-09-25 18:14:21,756::task::1217::Storage.TaskManager.Task::(stop) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::stopping in state preparing (force False) >Thread-71979::DEBUG::2014-09-25 18:14:21,756::task::993::Storage.TaskManager.Task::(_decref) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::ref 1 aborting True >Thread-71979::INFO::2014-09-25 18:14:21,756::task::1171::Storage.TaskManager.Task::(prepare) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::aborting: Task is aborted: u'' - code 100 >Thread-71979::DEBUG::2014-09-25 18:14:21,756::task::1176::Storage.TaskManager.Task::(prepare) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::Prepare: aborted: >Thread-71979::DEBUG::2014-09-25 18:14:21,757::task::993::Storage.TaskManager.Task::(_decref) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::ref 0 aborting True >Thread-71979::DEBUG::2014-09-25 18:14:21,757::task::928::Storage.TaskManager.Task::(_doAbort) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::Task._doAbort: force False >Thread-71979::DEBUG::2014-09-25 18:14:21,757::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71979::DEBUG::2014-09-25 18:14:21,757::task::595::Storage.TaskManager.Task::(_updateState) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::moving from state preparing -> state aborting >Thread-71979::DEBUG::2014-09-25 18:14:21,758::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::_aborting: recover policy none >Thread-71979::DEBUG::2014-09-25 18:14:21,758::task::595::Storage.TaskManager.Task::(_updateState) Task=`279ac599-e3b1-4e32-ba68-5a129638228f`::moving from state aborting -> state failed >Thread-71979::DEBUG::2014-09-25 18:14:21,758::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71979::DEBUG::2014-09-25 18:14:21,758::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71979::ERROR::2014-09-25 18:14:21,758::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71979::DEBUG::2014-09-25 18:14:21,759::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:23,116::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:23,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201426 s, 50.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:24,580::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:24,582::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71980::DEBUG::2014-09-25 18:14:24,584::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:25,170::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:25,205::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155026 s, 66.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:25,401::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:25,401::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:25,402::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:25,402::__init__::232::IOProcess::(_processLogs) (110863) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:25,404::__init__::232::IOProcess::(_processLogs) (110863) Queuing response >Thread-79::DEBUG::2014-09-25 18:14:25,404::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:14:25,426::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0113223 s, 49.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:25,427::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:25,428::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:25,428::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:25,428::__init__::232::IOProcess::(_processLogs) (110864) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:25,431::__init__::232::IOProcess::(_processLogs) (110864) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:25,432::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:25,432::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:25,433::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:25,433::__init__::232::IOProcess::(_processLogs) (110865) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:25,434::__init__::232::IOProcess::(_processLogs) (110865) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:25,434::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:25,435::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:25,435::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:25,435::__init__::232::IOProcess::(_processLogs) (110866) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:25,436::__init__::232::IOProcess::(_processLogs) (110866) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:25,436::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:25,436::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:25,437::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:25,437::__init__::232::IOProcess::(_processLogs) (110867) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:25,438::__init__::232::IOProcess::(_processLogs) (110867) Queuing response >Dummy-148::DEBUG::2014-09-25 18:14:27,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:27,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181979 s, 56.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:27,598::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:27,600::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71981::DEBUG::2014-09-25 18:14:27,601::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:14:28,891::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:28,891::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:28,892::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:28,892::__init__::232::IOProcess::(_processLogs) (110868) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:28,897::__init__::232::IOProcess::(_processLogs) (110868) Queuing response >Thread-81::DEBUG::2014-09-25 18:14:28,898::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:14:28,939::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0190357 s, 19.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:28,940::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:28,940::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:28,940::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:28,940::__init__::232::IOProcess::(_processLogs) (110869) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:28,948::__init__::232::IOProcess::(_processLogs) (110869) Queuing response >Dummy-148::DEBUG::2014-09-25 18:14:29,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:29,301::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144469 s, 70.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:30,616::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:30,617::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71982::DEBUG::2014-09-25 18:14:30,619::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:31,321::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:31,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178977 s, 57.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:31,785::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71983::DEBUG::2014-09-25 18:14:31,786::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:14:31,786::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71983::DEBUG::2014-09-25 18:14:31,789::task::595::Storage.TaskManager.Task::(_updateState) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::moving from state init -> state preparing >Thread-71983::INFO::2014-09-25 18:14:31,789::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71983::ERROR::2014-09-25 18:14:31,789::task::866::Storage.TaskManager.Task::(_setError) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71983::DEBUG::2014-09-25 18:14:31,790::task::885::Storage.TaskManager.Task::(_run) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::Task._run: 354285f3-de91-44c2-8143-3502b53c2d59 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71983::DEBUG::2014-09-25 18:14:31,790::task::1217::Storage.TaskManager.Task::(stop) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::stopping in state preparing (force False) >Thread-71983::DEBUG::2014-09-25 18:14:31,790::task::993::Storage.TaskManager.Task::(_decref) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::ref 1 aborting True >Thread-71983::INFO::2014-09-25 18:14:31,791::task::1171::Storage.TaskManager.Task::(prepare) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::aborting: Task is aborted: u'' - code 100 >Thread-71983::DEBUG::2014-09-25 18:14:31,791::task::1176::Storage.TaskManager.Task::(prepare) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::Prepare: aborted: >Thread-71983::DEBUG::2014-09-25 18:14:31,791::task::993::Storage.TaskManager.Task::(_decref) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::ref 0 aborting True >Thread-71983::DEBUG::2014-09-25 18:14:31,791::task::928::Storage.TaskManager.Task::(_doAbort) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::Task._doAbort: force False >Thread-71983::DEBUG::2014-09-25 18:14:31,791::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71983::DEBUG::2014-09-25 18:14:31,792::task::595::Storage.TaskManager.Task::(_updateState) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::moving from state preparing -> state aborting >Thread-71983::DEBUG::2014-09-25 18:14:31,792::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::_aborting: recover policy none >Thread-71983::DEBUG::2014-09-25 18:14:31,792::task::595::Storage.TaskManager.Task::(_updateState) Task=`354285f3-de91-44c2-8143-3502b53c2d59`::moving from state aborting -> state failed >Thread-71983::DEBUG::2014-09-25 18:14:31,792::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71983::DEBUG::2014-09-25 18:14:31,793::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71983::ERROR::2014-09-25 18:14:31,793::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71983::DEBUG::2014-09-25 18:14:31,793::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:33,373::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:33,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0220459 s, 46.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:33,632::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:33,634::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71984::DEBUG::2014-09-25 18:14:33,637::task::595::Storage.TaskManager.Task::(_updateState) Task=`5ef4e677-2b81-42e1-9823-be097b326ef1`::moving from state init -> state preparing >Thread-71984::INFO::2014-09-25 18:14:33,637::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71984::INFO::2014-09-25 18:14:33,638::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0113223', 'lastCheck': '8.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '25.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0190357', 'lastCheck': '4.7', 'valid': True}} >Thread-71984::DEBUG::2014-09-25 18:14:33,638::task::1191::Storage.TaskManager.Task::(prepare) Task=`5ef4e677-2b81-42e1-9823-be097b326ef1`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0113223', 'lastCheck': '8.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '25.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0190357', 'lastCheck': '4.7', 'valid': True}} >Thread-71984::DEBUG::2014-09-25 18:14:33,638::task::595::Storage.TaskManager.Task::(_updateState) Task=`5ef4e677-2b81-42e1-9823-be097b326ef1`::moving from state preparing -> state finished >Thread-71984::DEBUG::2014-09-25 18:14:33,638::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71984::DEBUG::2014-09-25 18:14:33,638::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71984::DEBUG::2014-09-25 18:14:33,639::task::993::Storage.TaskManager.Task::(_decref) Task=`5ef4e677-2b81-42e1-9823-be097b326ef1`::ref 0 aborting False >Thread-71984::DEBUG::2014-09-25 18:14:33,641::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:33,659::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:33,660::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71985::DEBUG::2014-09-25 18:14:33,662::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:35,431::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Thread-29::DEBUG::2014-09-25 18:14:35,440::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:35,441::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:35,441::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:35,441::__init__::232::IOProcess::(_processLogs) (110870) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:35,450::__init__::232::IOProcess::(_processLogs) (110870) Queuing response >Thread-79::DEBUG::2014-09-25 18:14:35,452::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:14:35,478::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0120057 s, 46.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:35,481::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:35,482::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:35,483::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:35,484::__init__::232::IOProcess::(_processLogs) (110871) Got request for method 'statvfs' >Dummy-148::DEBUG::2014-09-25 18:14:35,488::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0276608 s, 37.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:35,491::__init__::232::IOProcess::(_processLogs) (110871) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:35,494::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:35,496::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:35,497::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:35,497::__init__::232::IOProcess::(_processLogs) (110872) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:35,498::__init__::232::IOProcess::(_processLogs) (110872) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:35,500::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:35,501::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:35,501::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:35,502::__init__::232::IOProcess::(_processLogs) (110873) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:35,503::__init__::232::IOProcess::(_processLogs) (110873) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:35,505::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:35,506::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:35,507::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:35,508::__init__::232::IOProcess::(_processLogs) (110874) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:35,509::__init__::232::IOProcess::(_processLogs) (110874) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:36,709::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:36,710::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71986::DEBUG::2014-09-25 18:14:36,712::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:37,525::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:37,557::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0209718 s, 48.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:38,950::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:38,950::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:38,951::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:38,952::__init__::232::IOProcess::(_processLogs) (110875) Got request for method 'statvfs' >Thread-81::DEBUG::2014-09-25 18:14:38,952::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:14:38,953::__init__::232::IOProcess::(_processLogs) (110875) Queuing response >Thread-81::DEBUG::2014-09-25 18:14:38,962::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000281758 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:38,963::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:38,964::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:38,964::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:38,964::__init__::232::IOProcess::(_processLogs) (110876) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:38,965::__init__::232::IOProcess::(_processLogs) (110876) Queuing response >Dummy-148::DEBUG::2014-09-25 18:14:39,577::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:39,608::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016766 s, 61.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:39,726::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:39,728::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71987::DEBUG::2014-09-25 18:14:39,731::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:41,628::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:41,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0154649 s, 66.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:41,819::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:41,820::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71988::DEBUG::2014-09-25 18:14:41,821::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-71988::DEBUG::2014-09-25 18:14:41,823::task::595::Storage.TaskManager.Task::(_updateState) Task=`551ab964-a167-40e2-a783-0588b520db89`::moving from state init -> state preparing >Thread-71988::INFO::2014-09-25 18:14:41,823::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71988::ERROR::2014-09-25 18:14:41,824::task::866::Storage.TaskManager.Task::(_setError) Task=`551ab964-a167-40e2-a783-0588b520db89`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71988::DEBUG::2014-09-25 18:14:41,824::task::885::Storage.TaskManager.Task::(_run) Task=`551ab964-a167-40e2-a783-0588b520db89`::Task._run: 551ab964-a167-40e2-a783-0588b520db89 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71988::DEBUG::2014-09-25 18:14:41,824::task::1217::Storage.TaskManager.Task::(stop) Task=`551ab964-a167-40e2-a783-0588b520db89`::stopping in state preparing (force False) >Thread-71988::DEBUG::2014-09-25 18:14:41,825::task::993::Storage.TaskManager.Task::(_decref) Task=`551ab964-a167-40e2-a783-0588b520db89`::ref 1 aborting True >Thread-71988::INFO::2014-09-25 18:14:41,825::task::1171::Storage.TaskManager.Task::(prepare) Task=`551ab964-a167-40e2-a783-0588b520db89`::aborting: Task is aborted: u'' - code 100 >Thread-71988::DEBUG::2014-09-25 18:14:41,825::task::1176::Storage.TaskManager.Task::(prepare) Task=`551ab964-a167-40e2-a783-0588b520db89`::Prepare: aborted: >Thread-71988::DEBUG::2014-09-25 18:14:41,825::task::993::Storage.TaskManager.Task::(_decref) Task=`551ab964-a167-40e2-a783-0588b520db89`::ref 0 aborting True >Thread-71988::DEBUG::2014-09-25 18:14:41,825::task::928::Storage.TaskManager.Task::(_doAbort) Task=`551ab964-a167-40e2-a783-0588b520db89`::Task._doAbort: force False >Thread-71988::DEBUG::2014-09-25 18:14:41,826::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71988::DEBUG::2014-09-25 18:14:41,826::task::595::Storage.TaskManager.Task::(_updateState) Task=`551ab964-a167-40e2-a783-0588b520db89`::moving from state preparing -> state aborting >Thread-71988::DEBUG::2014-09-25 18:14:41,826::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`551ab964-a167-40e2-a783-0588b520db89`::_aborting: recover policy none >Thread-71988::DEBUG::2014-09-25 18:14:41,826::task::595::Storage.TaskManager.Task::(_updateState) Task=`551ab964-a167-40e2-a783-0588b520db89`::moving from state aborting -> state failed >Thread-71988::DEBUG::2014-09-25 18:14:41,827::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71988::DEBUG::2014-09-25 18:14:41,827::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71988::ERROR::2014-09-25 18:14:41,827::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71988::DEBUG::2014-09-25 18:14:41,827::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:42,747::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:42,748::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71989::DEBUG::2014-09-25 18:14:42,750::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:43,673::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:43,705::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151376 s, 67.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:45,511::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:45,512::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:45,512::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:45,512::__init__::232::IOProcess::(_processLogs) (110877) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:45,522::__init__::232::IOProcess::(_processLogs) (110877) Queuing response >Thread-79::DEBUG::2014-09-25 18:14:45,523::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:14:45,543::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00849534 s, 66.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:45,544::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:45,544::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:45,544::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:45,545::__init__::232::IOProcess::(_processLogs) (110878) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:45,550::__init__::232::IOProcess::(_processLogs) (110878) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:45,551::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:45,552::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:45,553::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:45,553::__init__::232::IOProcess::(_processLogs) (110879) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:45,553::__init__::232::IOProcess::(_processLogs) (110879) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:45,554::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:45,554::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:45,555::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:45,555::__init__::232::IOProcess::(_processLogs) (110880) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:45,555::__init__::232::IOProcess::(_processLogs) (110880) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:45,556::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:45,556::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:45,557::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:45,558::__init__::232::IOProcess::(_processLogs) (110881) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:45,558::__init__::232::IOProcess::(_processLogs) (110881) Queuing response >Dummy-148::DEBUG::2014-09-25 18:14:45,725::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:45,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152875 s, 67.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:45,767::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:45,770::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71990::DEBUG::2014-09-25 18:14:45,776::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:47,784::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:47,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166868 s, 61.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:48,792::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:48,794::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71991::DEBUG::2014-09-25 18:14:48,797::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f0e94fb-5bfd-44de-841d-7f2ea83b2670`::moving from state init -> state preparing >Thread-71991::INFO::2014-09-25 18:14:48,797::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71991::INFO::2014-09-25 18:14:48,798::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00849534', 'lastCheck': '3.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '40.8', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000281758', 'lastCheck': '9.8', 'valid': True}} >Thread-71991::DEBUG::2014-09-25 18:14:48,798::task::1191::Storage.TaskManager.Task::(prepare) Task=`2f0e94fb-5bfd-44de-841d-7f2ea83b2670`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00849534', 'lastCheck': '3.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '40.8', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000281758', 'lastCheck': '9.8', 'valid': True}} >Thread-71991::DEBUG::2014-09-25 18:14:48,798::task::595::Storage.TaskManager.Task::(_updateState) Task=`2f0e94fb-5bfd-44de-841d-7f2ea83b2670`::moving from state preparing -> state finished >Thread-71991::DEBUG::2014-09-25 18:14:48,798::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71991::DEBUG::2014-09-25 18:14:48,798::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71991::DEBUG::2014-09-25 18:14:48,799::task::993::Storage.TaskManager.Task::(_decref) Task=`2f0e94fb-5bfd-44de-841d-7f2ea83b2670`::ref 0 aborting False >Thread-71991::DEBUG::2014-09-25 18:14:48,801::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:48,817::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:48,818::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71992::DEBUG::2014-09-25 18:14:48,820::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:14:48,967::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:48,967::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:48,968::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:48,968::__init__::232::IOProcess::(_processLogs) (110882) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:48,968::__init__::232::IOProcess::(_processLogs) (110882) Queuing response >Thread-81::DEBUG::2014-09-25 18:14:48,969::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:14:48,988::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00304234 s, 122 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:48,989::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:48,989::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:48,989::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:48,990::__init__::232::IOProcess::(_processLogs) (110883) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:48,991::__init__::232::IOProcess::(_processLogs) (110883) Queuing response >Dummy-148::DEBUG::2014-09-25 18:14:49,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:49,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015164 s, 67.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:51,851::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71993::DEBUG::2014-09-25 18:14:51,852::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:14:51,852::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71993::DEBUG::2014-09-25 18:14:51,854::task::595::Storage.TaskManager.Task::(_updateState) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::moving from state init -> state preparing >Thread-71993::INFO::2014-09-25 18:14:51,855::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71993::ERROR::2014-09-25 18:14:51,855::task::866::Storage.TaskManager.Task::(_setError) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71993::DEBUG::2014-09-25 18:14:51,856::task::885::Storage.TaskManager.Task::(_run) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::Task._run: 3d636de4-f7b2-4c9e-ad55-dabf2e226310 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71993::DEBUG::2014-09-25 18:14:51,856::task::1217::Storage.TaskManager.Task::(stop) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::stopping in state preparing (force False) >Thread-71993::DEBUG::2014-09-25 18:14:51,856::task::993::Storage.TaskManager.Task::(_decref) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::ref 1 aborting True >Thread-71993::INFO::2014-09-25 18:14:51,856::task::1171::Storage.TaskManager.Task::(prepare) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::aborting: Task is aborted: u'' - code 100 >Thread-71993::DEBUG::2014-09-25 18:14:51,857::task::1176::Storage.TaskManager.Task::(prepare) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::Prepare: aborted: >Thread-71993::DEBUG::2014-09-25 18:14:51,857::task::993::Storage.TaskManager.Task::(_decref) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::ref 0 aborting True >Thread-71993::DEBUG::2014-09-25 18:14:51,857::task::928::Storage.TaskManager.Task::(_doAbort) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::Task._doAbort: force False >Thread-71993::DEBUG::2014-09-25 18:14:51,857::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71993::DEBUG::2014-09-25 18:14:51,857::task::595::Storage.TaskManager.Task::(_updateState) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::moving from state preparing -> state aborting >Thread-71993::DEBUG::2014-09-25 18:14:51,858::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::_aborting: recover policy none >Thread-71993::DEBUG::2014-09-25 18:14:51,858::task::595::Storage.TaskManager.Task::(_updateState) Task=`3d636de4-f7b2-4c9e-ad55-dabf2e226310`::moving from state aborting -> state failed >Thread-71993::DEBUG::2014-09-25 18:14:51,858::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71993::DEBUG::2014-09-25 18:14:51,858::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71993::ERROR::2014-09-25 18:14:51,859::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71993::DEBUG::2014-09-25 18:14:51,859::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:51,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:51,888::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:51,890::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71994::DEBUG::2014-09-25 18:14:51,892::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:51,907::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162931 s, 62.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:14:53,927::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:53,960::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.01642 s, 62.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:54,908::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:54,910::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71995::DEBUG::2014-09-25 18:14:54,912::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-71961::WARNING::2014-09-25 18:14:55,111::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-29::DEBUG::2014-09-25 18:14:55,558::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:55,558::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:55,559::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:55,559::__init__::232::IOProcess::(_processLogs) (110884) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:55,559::__init__::232::IOProcess::(_processLogs) (110884) Queuing response >Thread-79::DEBUG::2014-09-25 18:14:55,560::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:14:55,574::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00367616 s, 153 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:55,576::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:55,576::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:55,576::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:55,576::__init__::232::IOProcess::(_processLogs) (110885) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:55,579::__init__::232::IOProcess::(_processLogs) (110885) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:55,580::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:55,581::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:55,581::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:55,581::__init__::232::IOProcess::(_processLogs) (110886) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:55,582::__init__::232::IOProcess::(_processLogs) (110886) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:55,583::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:55,583::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:55,584::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:55,584::__init__::232::IOProcess::(_processLogs) (110887) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:55,588::__init__::232::IOProcess::(_processLogs) (110887) Queuing response >Thread-29::DEBUG::2014-09-25 18:14:55,589::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:55,589::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:55,589::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:55,590::__init__::232::IOProcess::(_processLogs) (110888) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:14:55,592::__init__::232::IOProcess::(_processLogs) (110888) Queuing response >Dummy-148::DEBUG::2014-09-25 18:14:55,980::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:56,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175284 s, 58.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:14:57,933::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:14:57,934::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71996::DEBUG::2014-09-25 18:14:57,936::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:14:58,027::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:14:58,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170218 s, 60.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:58,993::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:58,994::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:58,994::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:58,994::__init__::232::IOProcess::(_processLogs) (110889) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:58,994::__init__::232::IOProcess::(_processLogs) (110889) Queuing response >Thread-81::DEBUG::2014-09-25 18:14:58,996::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:14:59,004::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000424399 s, 874 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:14:59,006::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:14:59,006::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:14:59,006::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:14:59,007::__init__::232::IOProcess::(_processLogs) (110890) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:14:59,007::__init__::232::IOProcess::(_processLogs) (110890) Queuing response >Dummy-148::DEBUG::2014-09-25 18:15:00,073::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:00,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.019819 s, 51.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:00,959::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:00,962::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71997::DEBUG::2014-09-25 18:15:00,966::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:01,883::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-71998::DEBUG::2014-09-25 18:15:01,884::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:15:01,884::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71998::DEBUG::2014-09-25 18:15:01,887::task::595::Storage.TaskManager.Task::(_updateState) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::moving from state init -> state preparing >Thread-71998::INFO::2014-09-25 18:15:01,887::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-71998::ERROR::2014-09-25 18:15:01,887::task::866::Storage.TaskManager.Task::(_setError) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-71998::DEBUG::2014-09-25 18:15:01,888::task::885::Storage.TaskManager.Task::(_run) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::Task._run: 8e6704dd-dea9-4e71-81ab-20b624e1b7fe (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-71998::DEBUG::2014-09-25 18:15:01,888::task::1217::Storage.TaskManager.Task::(stop) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::stopping in state preparing (force False) >Thread-71998::DEBUG::2014-09-25 18:15:01,888::task::993::Storage.TaskManager.Task::(_decref) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::ref 1 aborting True >Thread-71998::INFO::2014-09-25 18:15:01,889::task::1171::Storage.TaskManager.Task::(prepare) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::aborting: Task is aborted: u'' - code 100 >Thread-71998::DEBUG::2014-09-25 18:15:01,889::task::1176::Storage.TaskManager.Task::(prepare) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::Prepare: aborted: >Thread-71998::DEBUG::2014-09-25 18:15:01,889::task::993::Storage.TaskManager.Task::(_decref) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::ref 0 aborting True >Thread-71998::DEBUG::2014-09-25 18:15:01,889::task::928::Storage.TaskManager.Task::(_doAbort) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::Task._doAbort: force False >Thread-71998::DEBUG::2014-09-25 18:15:01,890::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71998::DEBUG::2014-09-25 18:15:01,890::task::595::Storage.TaskManager.Task::(_updateState) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::moving from state preparing -> state aborting >Thread-71998::DEBUG::2014-09-25 18:15:01,890::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::_aborting: recover policy none >Thread-71998::DEBUG::2014-09-25 18:15:01,890::task::595::Storage.TaskManager.Task::(_updateState) Task=`8e6704dd-dea9-4e71-81ab-20b624e1b7fe`::moving from state aborting -> state failed >Thread-71998::DEBUG::2014-09-25 18:15:01,890::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71998::DEBUG::2014-09-25 18:15:01,891::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71998::ERROR::2014-09-25 18:15:01,891::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-71998::DEBUG::2014-09-25 18:15:01,891::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:02,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:02,153::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0187711 s, 54.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:03,981::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:03,982::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-71999::DEBUG::2014-09-25 18:15:03,985::task::595::Storage.TaskManager.Task::(_updateState) Task=`4d7affed-3c50-4ed7-af08-bb57a0beedc0`::moving from state init -> state preparing >Thread-71999::INFO::2014-09-25 18:15:03,985::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-71999::INFO::2014-09-25 18:15:03,986::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00367616', 'lastCheck': '8.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '56.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000424399', 'lastCheck': '5.0', 'valid': True}} >Thread-71999::DEBUG::2014-09-25 18:15:03,986::task::1191::Storage.TaskManager.Task::(prepare) Task=`4d7affed-3c50-4ed7-af08-bb57a0beedc0`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00367616', 'lastCheck': '8.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '56.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000424399', 'lastCheck': '5.0', 'valid': True}} >Thread-71999::DEBUG::2014-09-25 18:15:03,986::task::595::Storage.TaskManager.Task::(_updateState) Task=`4d7affed-3c50-4ed7-af08-bb57a0beedc0`::moving from state preparing -> state finished >Thread-71999::DEBUG::2014-09-25 18:15:03,986::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-71999::DEBUG::2014-09-25 18:15:03,987::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-71999::DEBUG::2014-09-25 18:15:03,987::task::993::Storage.TaskManager.Task::(_decref) Task=`4d7affed-3c50-4ed7-af08-bb57a0beedc0`::ref 0 aborting False >Thread-71999::DEBUG::2014-09-25 18:15:03,989::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:04,005::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:04,006::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72000::DEBUG::2014-09-25 18:15:04,008::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:04,173::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:04,195::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.01317 s, 77.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:05,200::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72001::DEBUG::2014-09-25 18:15:05,201::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.connectStorageServer' in bridge with {u'connectionParams': [{u'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8', u'connection': u'pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', u'iqn': u'', u'user': u'', u'tpgt': u'', u'password': u'', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1} >JsonRpcServer::DEBUG::2014-09-25 18:15:05,201::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72001::DEBUG::2014-09-25 18:15:05,203::task::595::Storage.TaskManager.Task::(_updateState) Task=`ecd90552-46d7-4a12-84cb-d10cac117193`::moving from state init -> state preparing >Thread-72001::INFO::2014-09-25 18:15:05,204::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'connection': u'pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', u'iqn': u'', u'user': u'', u'tpgt': u'', u'password': '******', u'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8', u'port': u''}], options=None) >Thread-72001::DEBUG::2014-09-25 18:15:05,207::hsm::2372::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Thread-29::DEBUG::2014-09-25 18:15:05,208::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:05,209::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:05,209::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:05,209::__init__::232::IOProcess::(_processLogs) (110891) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:15:05,593::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:05,593::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:05,594::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:05,594::__init__::232::IOProcess::(_processLogs) (110892) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:05,594::__init__::232::IOProcess::(_processLogs) (110892) Queuing response >Thread-79::DEBUG::2014-09-25 18:15:05,595::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:15:05,611::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00341075 s, 165 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:05,612::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:05,613::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:05,613::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:05,613::__init__::232::IOProcess::(_processLogs) (110893) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:05,615::__init__::232::IOProcess::(_processLogs) (110893) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:05,616::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:05,616::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:05,617::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:05,617::__init__::232::IOProcess::(_processLogs) (110894) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:05,617::__init__::232::IOProcess::(_processLogs) (110894) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:05,618::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:05,618::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:05,618::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:05,619::__init__::232::IOProcess::(_processLogs) (110895) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:05,619::__init__::232::IOProcess::(_processLogs) (110895) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:05,620::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:05,620::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:05,620::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:05,621::__init__::232::IOProcess::(_processLogs) (110896) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:05,621::__init__::232::IOProcess::(_processLogs) (110896) Queuing response >Dummy-148::DEBUG::2014-09-25 18:15:06,214::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:06,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143667 s, 71.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:07,055::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:07,057::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72002::DEBUG::2014-09-25 18:15:07,059::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:08,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:08,295::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179005 s, 57.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:09,009::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:09,010::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:09,010::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:09,010::__init__::232::IOProcess::(_processLogs) (110897) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:09,018::__init__::232::IOProcess::(_processLogs) (110897) Queuing response >Thread-81::DEBUG::2014-09-25 18:15:09,019::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:15:09,039::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00137427 s, 270 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:09,040::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:09,040::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:09,040::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:09,041::__init__::232::IOProcess::(_processLogs) (110898) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:09,042::__init__::232::IOProcess::(_processLogs) (110898) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:10,073::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:10,075::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72003::DEBUG::2014-09-25 18:15:10,077::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:10,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:10,344::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0175489 s, 58.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:11,921::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72004::DEBUG::2014-09-25 18:15:11,922::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:15:11,922::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72004::DEBUG::2014-09-25 18:15:11,925::task::595::Storage.TaskManager.Task::(_updateState) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::moving from state init -> state preparing >Thread-72004::INFO::2014-09-25 18:15:11,925::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72004::ERROR::2014-09-25 18:15:11,925::task::866::Storage.TaskManager.Task::(_setError) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72004::DEBUG::2014-09-25 18:15:11,926::task::885::Storage.TaskManager.Task::(_run) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::Task._run: 460fce4b-98f2-4d59-895e-23a4c44ad646 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72004::DEBUG::2014-09-25 18:15:11,926::task::1217::Storage.TaskManager.Task::(stop) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::stopping in state preparing (force False) >Thread-72004::DEBUG::2014-09-25 18:15:11,926::task::993::Storage.TaskManager.Task::(_decref) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::ref 1 aborting True >Thread-72004::INFO::2014-09-25 18:15:11,927::task::1171::Storage.TaskManager.Task::(prepare) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::aborting: Task is aborted: u'' - code 100 >Thread-72004::DEBUG::2014-09-25 18:15:11,927::task::1176::Storage.TaskManager.Task::(prepare) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::Prepare: aborted: >Thread-72004::DEBUG::2014-09-25 18:15:11,927::task::993::Storage.TaskManager.Task::(_decref) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::ref 0 aborting True >Thread-72004::DEBUG::2014-09-25 18:15:11,927::task::928::Storage.TaskManager.Task::(_doAbort) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::Task._doAbort: force False >Thread-72004::DEBUG::2014-09-25 18:15:11,928::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72004::DEBUG::2014-09-25 18:15:11,928::task::595::Storage.TaskManager.Task::(_updateState) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::moving from state preparing -> state aborting >Thread-72004::DEBUG::2014-09-25 18:15:11,928::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::_aborting: recover policy none >Thread-72004::DEBUG::2014-09-25 18:15:11,928::task::595::Storage.TaskManager.Task::(_updateState) Task=`460fce4b-98f2-4d59-895e-23a4c44ad646`::moving from state aborting -> state failed >Thread-72004::DEBUG::2014-09-25 18:15:11,928::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72004::DEBUG::2014-09-25 18:15:11,929::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72004::ERROR::2014-09-25 18:15:11,929::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72004::DEBUG::2014-09-25 18:15:11,929::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:12,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:12,386::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0129519 s, 79.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:13,091::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:13,092::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72005::DEBUG::2014-09-25 18:15:13,094::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:14,406::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:14,436::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166295 s, 61.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:15,622::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:15,623::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:15,623::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:15,623::__init__::232::IOProcess::(_processLogs) (110899) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:15,625::__init__::232::IOProcess::(_processLogs) (110899) Queuing response >Thread-79::DEBUG::2014-09-25 18:15:15,626::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:15:15,643::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0025918 s, 217 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:15,644::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:15,644::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:15,644::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:15,645::__init__::232::IOProcess::(_processLogs) (110900) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:15,646::__init__::232::IOProcess::(_processLogs) (110900) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:15,647::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:15,647::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:15,648::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:15,648::__init__::232::IOProcess::(_processLogs) (110901) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:15,649::__init__::232::IOProcess::(_processLogs) (110901) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:15,649::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:15,650::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:15,650::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:15,651::__init__::232::IOProcess::(_processLogs) (110902) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:15,651::__init__::232::IOProcess::(_processLogs) (110902) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:15,651::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:15,652::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:15,653::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:15,653::__init__::232::IOProcess::(_processLogs) (110903) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:15,654::__init__::232::IOProcess::(_processLogs) (110903) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:16,108::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:16,111::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72006::DEBUG::2014-09-25 18:15:16,117::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:16,457::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:16,491::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0225313 s, 45.4 MB/s\n'; <rc> = 0 >Thread-71974::WARNING::2014-09-25 18:15:18,031::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:15:18,032::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:15:18,032::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Dummy-148::DEBUG::2014-09-25 18:15:18,674::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:18,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145111 s, 70.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:19,044::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:19,044::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:19,045::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:19,045::__init__::232::IOProcess::(_processLogs) (110904) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:19,046::__init__::232::IOProcess::(_processLogs) (110904) Queuing response >Thread-81::DEBUG::2014-09-25 18:15:19,047::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:15:19,071::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00858087 s, 43.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:19,072::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:19,072::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:19,073::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:19,073::__init__::232::IOProcess::(_processLogs) (110905) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:19,080::__init__::232::IOProcess::(_processLogs) (110905) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:19,129::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:19,131::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72007::DEBUG::2014-09-25 18:15:19,133::task::595::Storage.TaskManager.Task::(_updateState) Task=`0fb23b28-b552-4f83-9f66-528f276aa89d`::moving from state init -> state preparing >Thread-72007::INFO::2014-09-25 18:15:19,134::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72007::INFO::2014-09-25 18:15:19,134::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0025918', 'lastCheck': '3.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.1', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00858087', 'lastCheck': '0.1', 'valid': True}} >Thread-72007::DEBUG::2014-09-25 18:15:19,135::task::1191::Storage.TaskManager.Task::(prepare) Task=`0fb23b28-b552-4f83-9f66-528f276aa89d`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0025918', 'lastCheck': '3.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '1.1', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00858087', 'lastCheck': '0.1', 'valid': True}} >Thread-72007::DEBUG::2014-09-25 18:15:19,135::task::595::Storage.TaskManager.Task::(_updateState) Task=`0fb23b28-b552-4f83-9f66-528f276aa89d`::moving from state preparing -> state finished >Thread-72007::DEBUG::2014-09-25 18:15:19,135::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72007::DEBUG::2014-09-25 18:15:19,135::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72007::DEBUG::2014-09-25 18:15:19,135::task::993::Storage.TaskManager.Task::(_decref) Task=`0fb23b28-b552-4f83-9f66-528f276aa89d`::ref 0 aborting False >Thread-72007::DEBUG::2014-09-25 18:15:19,138::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:19,187::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:19,189::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72008::DEBUG::2014-09-25 18:15:19,190::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:20,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:20,750::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.02083 s, 49.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:21,973::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72009::DEBUG::2014-09-25 18:15:21,974::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:15:21,974::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72009::DEBUG::2014-09-25 18:15:21,977::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::moving from state init -> state preparing >Thread-72009::INFO::2014-09-25 18:15:21,977::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72009::ERROR::2014-09-25 18:15:21,977::task::866::Storage.TaskManager.Task::(_setError) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72009::DEBUG::2014-09-25 18:15:21,978::task::885::Storage.TaskManager.Task::(_run) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::Task._run: 0c321947-5705-4066-b20f-a7f2a466cbba (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72009::DEBUG::2014-09-25 18:15:21,978::task::1217::Storage.TaskManager.Task::(stop) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::stopping in state preparing (force False) >Thread-72009::DEBUG::2014-09-25 18:15:21,978::task::993::Storage.TaskManager.Task::(_decref) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::ref 1 aborting True >Thread-72009::INFO::2014-09-25 18:15:21,979::task::1171::Storage.TaskManager.Task::(prepare) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::aborting: Task is aborted: u'' - code 100 >Thread-72009::DEBUG::2014-09-25 18:15:21,979::task::1176::Storage.TaskManager.Task::(prepare) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::Prepare: aborted: >Thread-72009::DEBUG::2014-09-25 18:15:21,979::task::993::Storage.TaskManager.Task::(_decref) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::ref 0 aborting True >Thread-72009::DEBUG::2014-09-25 18:15:21,979::task::928::Storage.TaskManager.Task::(_doAbort) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::Task._doAbort: force False >Thread-72009::DEBUG::2014-09-25 18:15:21,980::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72009::DEBUG::2014-09-25 18:15:21,980::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::moving from state preparing -> state aborting >Thread-72009::DEBUG::2014-09-25 18:15:21,980::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::_aborting: recover policy none >Thread-72009::DEBUG::2014-09-25 18:15:21,980::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c321947-5705-4066-b20f-a7f2a466cbba`::moving from state aborting -> state failed >Thread-72009::DEBUG::2014-09-25 18:15:21,980::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72009::DEBUG::2014-09-25 18:15:21,981::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72009::ERROR::2014-09-25 18:15:21,981::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72009::DEBUG::2014-09-25 18:15:21,981::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:22,255::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:22,256::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72010::DEBUG::2014-09-25 18:15:22,258::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:22,769::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:22,796::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165886 s, 61.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:15:24,816::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:24,846::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166477 s, 61.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:25,326::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:25,328::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72011::DEBUG::2014-09-25 18:15:25,330::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:15:25,654::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:25,654::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:25,655::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:25,655::__init__::232::IOProcess::(_processLogs) (110906) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:25,655::__init__::232::IOProcess::(_processLogs) (110906) Queuing response >Thread-79::DEBUG::2014-09-25 18:15:25,657::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:15:25,665::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000465587 s, 1.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:25,667::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:25,667::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:25,667::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:25,668::__init__::232::IOProcess::(_processLogs) (110907) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:25,668::__init__::232::IOProcess::(_processLogs) (110907) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:25,670::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:25,670::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:25,671::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:25,671::__init__::232::IOProcess::(_processLogs) (110908) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:25,671::__init__::232::IOProcess::(_processLogs) (110908) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:25,672::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:25,672::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:25,672::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:25,672::__init__::232::IOProcess::(_processLogs) (110909) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:25,673::__init__::232::IOProcess::(_processLogs) (110909) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:25,673::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:25,674::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:25,674::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:25,674::__init__::232::IOProcess::(_processLogs) (110910) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:25,674::__init__::232::IOProcess::(_processLogs) (110910) Queuing response >Dummy-148::DEBUG::2014-09-25 18:15:26,865::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:26,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155007 s, 66.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:28,037::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:28,037::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:28,037::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:28,038::__init__::232::IOProcess::(_processLogs) (110911) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:15:28,038::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:28,038::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:28,038::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:28,039::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:28,039::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:28,039::__init__::232::IOProcess::(_processLogs) (110912) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:15:28,039::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:28,040::__init__::232::IOProcess::(_processLogs) (110913) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:15:28,040::__init__::232::IOProcess::(_processLogs) (110911) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:28,044::__init__::232::IOProcess::(_processLogs) (110913) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:28,346::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:28,348::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72015::DEBUG::2014-09-25 18:15:28,350::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:28,913::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:28,941::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164425 s, 62.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:29,081::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:29,082::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:29,082::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:29,082::__init__::232::IOProcess::(_processLogs) (110914) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:29,088::__init__::232::IOProcess::(_processLogs) (110914) Queuing response >Thread-81::DEBUG::2014-09-25 18:15:29,089::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:15:29,109::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00470507 s, 78.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:29,111::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:29,111::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:29,111::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:29,112::__init__::232::IOProcess::(_processLogs) (110915) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:29,117::__init__::232::IOProcess::(_processLogs) (110915) Queuing response >Dummy-148::DEBUG::2014-09-25 18:15:30,961::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:31,363::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:31,365::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72016::DEBUG::2014-09-25 18:15:31,366::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:31,669::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.24551 s, 4.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:32,108::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72017::DEBUG::2014-09-25 18:15:32,109::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:15:32,109::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72017::DEBUG::2014-09-25 18:15:32,111::task::595::Storage.TaskManager.Task::(_updateState) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::moving from state init -> state preparing >Thread-72017::INFO::2014-09-25 18:15:32,112::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72017::ERROR::2014-09-25 18:15:32,112::task::866::Storage.TaskManager.Task::(_setError) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72017::DEBUG::2014-09-25 18:15:32,113::task::885::Storage.TaskManager.Task::(_run) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::Task._run: 08449e42-b3c3-4d66-a1df-3b921c04a445 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72017::DEBUG::2014-09-25 18:15:32,113::task::1217::Storage.TaskManager.Task::(stop) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::stopping in state preparing (force False) >Thread-72017::DEBUG::2014-09-25 18:15:32,113::task::993::Storage.TaskManager.Task::(_decref) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::ref 1 aborting True >Thread-72017::INFO::2014-09-25 18:15:32,113::task::1171::Storage.TaskManager.Task::(prepare) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::aborting: Task is aborted: u'' - code 100 >Thread-72017::DEBUG::2014-09-25 18:15:32,114::task::1176::Storage.TaskManager.Task::(prepare) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::Prepare: aborted: >Thread-72017::DEBUG::2014-09-25 18:15:32,114::task::993::Storage.TaskManager.Task::(_decref) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::ref 0 aborting True >Thread-72017::DEBUG::2014-09-25 18:15:32,114::task::928::Storage.TaskManager.Task::(_doAbort) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::Task._doAbort: force False >Thread-72017::DEBUG::2014-09-25 18:15:32,114::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72017::DEBUG::2014-09-25 18:15:32,114::task::595::Storage.TaskManager.Task::(_updateState) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::moving from state preparing -> state aborting >Thread-72017::DEBUG::2014-09-25 18:15:32,115::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::_aborting: recover policy none >Thread-72017::DEBUG::2014-09-25 18:15:32,115::task::595::Storage.TaskManager.Task::(_updateState) Task=`08449e42-b3c3-4d66-a1df-3b921c04a445`::moving from state aborting -> state failed >Thread-72017::DEBUG::2014-09-25 18:15:32,115::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72017::DEBUG::2014-09-25 18:15:32,115::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72017::ERROR::2014-09-25 18:15:32,115::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72017::DEBUG::2014-09-25 18:15:32,116::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:33,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:33,715::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0180566 s, 56.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:34,380::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:34,381::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72018::DEBUG::2014-09-25 18:15:34,384::task::595::Storage.TaskManager.Task::(_updateState) Task=`165dbd56-2fd8-451c-821c-416494c7cd6e`::moving from state init -> state preparing >Thread-72018::INFO::2014-09-25 18:15:34,384::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72018::INFO::2014-09-25 18:15:34,385::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000465587', 'lastCheck': '8.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '16.4', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00470507', 'lastCheck': '5.3', 'valid': True}} >Thread-72018::DEBUG::2014-09-25 18:15:34,385::task::1191::Storage.TaskManager.Task::(prepare) Task=`165dbd56-2fd8-451c-821c-416494c7cd6e`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000465587', 'lastCheck': '8.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '16.4', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00470507', 'lastCheck': '5.3', 'valid': True}} >Thread-72018::DEBUG::2014-09-25 18:15:34,385::task::595::Storage.TaskManager.Task::(_updateState) Task=`165dbd56-2fd8-451c-821c-416494c7cd6e`::moving from state preparing -> state finished >Thread-72018::DEBUG::2014-09-25 18:15:34,385::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72018::DEBUG::2014-09-25 18:15:34,386::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72018::DEBUG::2014-09-25 18:15:34,386::task::993::Storage.TaskManager.Task::(_decref) Task=`165dbd56-2fd8-451c-821c-416494c7cd6e`::ref 0 aborting False >Thread-72018::DEBUG::2014-09-25 18:15:34,388::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:34,415::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:34,416::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72019::DEBUG::2014-09-25 18:15:34,418::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:15:35,676::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:35,676::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:35,677::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:35,677::__init__::232::IOProcess::(_processLogs) (110916) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:35,682::__init__::232::IOProcess::(_processLogs) (110916) Queuing response >Thread-79::DEBUG::2014-09-25 18:15:35,683::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:15:35,705::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0072841 s, 77.2 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:35,706::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:35,707::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:35,707::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:35,707::__init__::232::IOProcess::(_processLogs) (110917) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:35,714::__init__::232::IOProcess::(_processLogs) (110917) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:35,715::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:35,715::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:35,716::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:35,716::__init__::232::IOProcess::(_processLogs) (110918) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:35,717::__init__::232::IOProcess::(_processLogs) (110918) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:35,717::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:35,718::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:35,718::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:35,718::__init__::232::IOProcess::(_processLogs) (110919) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:35,719::__init__::232::IOProcess::(_processLogs) (110919) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:35,719::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:35,720::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:35,721::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:35,721::__init__::232::IOProcess::(_processLogs) (110920) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:35,721::__init__::232::IOProcess::(_processLogs) (110920) Queuing response >Dummy-148::DEBUG::2014-09-25 18:15:35,735::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:35,833::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0293572 s, 34.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:37,504::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:37,506::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72020::DEBUG::2014-09-25 18:15:37,508::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:37,853::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:37,882::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017327 s, 59.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:39,119::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:39,119::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:39,119::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:39,120::__init__::232::IOProcess::(_processLogs) (110921) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:39,121::__init__::232::IOProcess::(_processLogs) (110921) Queuing response >Thread-81::DEBUG::2014-09-25 18:15:39,122::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:15:39,140::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00251356 s, 148 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:39,142::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:39,142::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:39,142::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:39,142::__init__::232::IOProcess::(_processLogs) (110922) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:39,144::__init__::232::IOProcess::(_processLogs) (110922) Queuing response >Dummy-148::DEBUG::2014-09-25 18:15:39,902::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:39,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132674 s, 77.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:40,521::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:40,522::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72021::DEBUG::2014-09-25 18:15:40,524::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:41,944::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:41,972::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0178507 s, 57.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:42,139::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:42,141::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72022::DEBUG::2014-09-25 18:15:42,141::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72022::DEBUG::2014-09-25 18:15:42,143::task::595::Storage.TaskManager.Task::(_updateState) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::moving from state init -> state preparing >Thread-72022::INFO::2014-09-25 18:15:42,143::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72022::ERROR::2014-09-25 18:15:42,144::task::866::Storage.TaskManager.Task::(_setError) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72022::DEBUG::2014-09-25 18:15:42,144::task::885::Storage.TaskManager.Task::(_run) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::Task._run: e22f4c58-965f-413b-a59c-a3d83de3a0b7 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72022::DEBUG::2014-09-25 18:15:42,145::task::1217::Storage.TaskManager.Task::(stop) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::stopping in state preparing (force False) >Thread-72022::DEBUG::2014-09-25 18:15:42,145::task::993::Storage.TaskManager.Task::(_decref) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::ref 1 aborting True >Thread-72022::INFO::2014-09-25 18:15:42,145::task::1171::Storage.TaskManager.Task::(prepare) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::aborting: Task is aborted: u'' - code 100 >Thread-72022::DEBUG::2014-09-25 18:15:42,145::task::1176::Storage.TaskManager.Task::(prepare) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::Prepare: aborted: >Thread-72022::DEBUG::2014-09-25 18:15:42,146::task::993::Storage.TaskManager.Task::(_decref) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::ref 0 aborting True >Thread-72022::DEBUG::2014-09-25 18:15:42,146::task::928::Storage.TaskManager.Task::(_doAbort) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::Task._doAbort: force False >Thread-72022::DEBUG::2014-09-25 18:15:42,146::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72022::DEBUG::2014-09-25 18:15:42,146::task::595::Storage.TaskManager.Task::(_updateState) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::moving from state preparing -> state aborting >Thread-72022::DEBUG::2014-09-25 18:15:42,146::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::_aborting: recover policy none >Thread-72022::DEBUG::2014-09-25 18:15:42,147::task::595::Storage.TaskManager.Task::(_updateState) Task=`e22f4c58-965f-413b-a59c-a3d83de3a0b7`::moving from state aborting -> state failed >Thread-72022::DEBUG::2014-09-25 18:15:42,147::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72022::DEBUG::2014-09-25 18:15:42,147::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72022::ERROR::2014-09-25 18:15:42,147::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72022::DEBUG::2014-09-25 18:15:42,148::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:43,538::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:43,539::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72023::DEBUG::2014-09-25 18:15:43,541::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:43,992::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:44,022::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164847 s, 62.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:45,721::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:45,722::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:45,722::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:45,722::__init__::232::IOProcess::(_processLogs) (110923) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:45,723::__init__::232::IOProcess::(_processLogs) (110923) Queuing response >Thread-79::DEBUG::2014-09-25 18:15:45,725::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:15:45,744::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00231419 s, 243 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:45,745::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:45,745::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:45,745::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:45,746::__init__::232::IOProcess::(_processLogs) (110924) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:45,753::__init__::232::IOProcess::(_processLogs) (110924) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:45,755::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:45,755::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:45,755::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:45,755::__init__::232::IOProcess::(_processLogs) (110925) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:45,756::__init__::232::IOProcess::(_processLogs) (110925) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:45,757::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:45,757::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:45,757::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:45,757::__init__::232::IOProcess::(_processLogs) (110926) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:45,758::__init__::232::IOProcess::(_processLogs) (110926) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:45,758::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:45,759::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:45,759::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:45,759::__init__::232::IOProcess::(_processLogs) (110927) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:45,760::__init__::232::IOProcess::(_processLogs) (110927) Queuing response >Dummy-148::DEBUG::2014-09-25 18:15:46,043::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:46,086::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.025924 s, 39.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:46,556::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:46,559::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72024::DEBUG::2014-09-25 18:15:46,565::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:48,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:48,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170706 s, 60.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:49,147::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:49,148::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:49,149::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:49,149::__init__::232::IOProcess::(_processLogs) (110928) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:49,151::__init__::232::IOProcess::(_processLogs) (110928) Queuing response >Thread-81::DEBUG::2014-09-25 18:15:49,154::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:15:49,173::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00549842 s, 67.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:49,174::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:49,174::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:49,175::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:49,175::__init__::232::IOProcess::(_processLogs) (110929) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:49,176::__init__::232::IOProcess::(_processLogs) (110929) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:49,580::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:49,581::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72025::DEBUG::2014-09-25 18:15:49,584::task::595::Storage.TaskManager.Task::(_updateState) Task=`1d6af64c-47ec-4afb-87ab-ea856e90f795`::moving from state init -> state preparing >Thread-72025::INFO::2014-09-25 18:15:49,584::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72025::INFO::2014-09-25 18:15:49,584::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00231419', 'lastCheck': '3.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '31.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00549842', 'lastCheck': '0.4', 'valid': True}} >Thread-72025::DEBUG::2014-09-25 18:15:49,585::task::1191::Storage.TaskManager.Task::(prepare) Task=`1d6af64c-47ec-4afb-87ab-ea856e90f795`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00231419', 'lastCheck': '3.8', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '31.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00549842', 'lastCheck': '0.4', 'valid': True}} >Thread-72025::DEBUG::2014-09-25 18:15:49,585::task::595::Storage.TaskManager.Task::(_updateState) Task=`1d6af64c-47ec-4afb-87ab-ea856e90f795`::moving from state preparing -> state finished >Thread-72025::DEBUG::2014-09-25 18:15:49,585::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72025::DEBUG::2014-09-25 18:15:49,585::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72025::DEBUG::2014-09-25 18:15:49,586::task::993::Storage.TaskManager.Task::(_decref) Task=`1d6af64c-47ec-4afb-87ab-ea856e90f795`::ref 0 aborting False >Thread-72025::DEBUG::2014-09-25 18:15:49,588::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:49,617::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:49,618::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72026::DEBUG::2014-09-25 18:15:49,620::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:50,322::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:50,353::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017308 s, 59.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:52,172::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:52,173::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72027::DEBUG::2014-09-25 18:15:52,174::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72027::DEBUG::2014-09-25 18:15:52,175::task::595::Storage.TaskManager.Task::(_updateState) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::moving from state init -> state preparing >Thread-72027::INFO::2014-09-25 18:15:52,176::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72027::ERROR::2014-09-25 18:15:52,176::task::866::Storage.TaskManager.Task::(_setError) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72027::DEBUG::2014-09-25 18:15:52,177::task::885::Storage.TaskManager.Task::(_run) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::Task._run: 6fdaae3a-4831-455c-8d3e-65fe391b4979 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72027::DEBUG::2014-09-25 18:15:52,177::task::1217::Storage.TaskManager.Task::(stop) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::stopping in state preparing (force False) >Thread-72027::DEBUG::2014-09-25 18:15:52,177::task::993::Storage.TaskManager.Task::(_decref) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::ref 1 aborting True >Thread-72027::INFO::2014-09-25 18:15:52,178::task::1171::Storage.TaskManager.Task::(prepare) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::aborting: Task is aborted: u'' - code 100 >Thread-72027::DEBUG::2014-09-25 18:15:52,178::task::1176::Storage.TaskManager.Task::(prepare) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::Prepare: aborted: >Thread-72027::DEBUG::2014-09-25 18:15:52,178::task::993::Storage.TaskManager.Task::(_decref) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::ref 0 aborting True >Thread-72027::DEBUG::2014-09-25 18:15:52,179::task::928::Storage.TaskManager.Task::(_doAbort) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::Task._doAbort: force False >Thread-72027::DEBUG::2014-09-25 18:15:52,179::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72027::DEBUG::2014-09-25 18:15:52,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::moving from state preparing -> state aborting >Thread-72027::DEBUG::2014-09-25 18:15:52,179::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::_aborting: recover policy none >Thread-72027::DEBUG::2014-09-25 18:15:52,179::task::595::Storage.TaskManager.Task::(_updateState) Task=`6fdaae3a-4831-455c-8d3e-65fe391b4979`::moving from state aborting -> state failed >Thread-72027::DEBUG::2014-09-25 18:15:52,180::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72027::DEBUG::2014-09-25 18:15:52,180::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72027::ERROR::2014-09-25 18:15:52,180::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72027::DEBUG::2014-09-25 18:15:52,181::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:52,372::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:52,405::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181044 s, 56.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:52,674::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:52,676::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72028::DEBUG::2014-09-25 18:15:52,678::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:15:54,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:54,447::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132151 s, 77.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:55,691::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:55,693::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72029::DEBUG::2014-09-25 18:15:55,694::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:15:55,762::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:55,762::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:55,763::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-79::DEBUG::2014-09-25 18:15:55,764::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:15:55,765::__init__::232::IOProcess::(_processLogs) (110930) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:55,773::__init__::232::IOProcess::(_processLogs) (110930) Queuing response >Thread-79::DEBUG::2014-09-25 18:15:55,774::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000385713 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:55,775::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:55,776::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:55,776::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:55,776::__init__::232::IOProcess::(_processLogs) (110931) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:55,777::__init__::232::IOProcess::(_processLogs) (110931) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:55,778::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:55,778::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:55,778::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:55,779::__init__::232::IOProcess::(_processLogs) (110932) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:55,779::__init__::232::IOProcess::(_processLogs) (110932) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:55,780::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:55,780::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:55,780::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:55,781::__init__::232::IOProcess::(_processLogs) (110933) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:55,781::__init__::232::IOProcess::(_processLogs) (110933) Queuing response >Thread-29::DEBUG::2014-09-25 18:15:55,782::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:55,782::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:55,782::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:55,783::__init__::232::IOProcess::(_processLogs) (110934) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:15:55,783::__init__::232::IOProcess::(_processLogs) (110934) Queuing response >Dummy-148::DEBUG::2014-09-25 18:15:56,467::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:56,496::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182663 s, 56.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:15:58,516::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:15:58,570::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0287643 s, 35.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:15:58,711::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:15:58,713::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72030::DEBUG::2014-09-25 18:15:58,715::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:15:59,178::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:59,178::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:59,179::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:59,179::__init__::232::IOProcess::(_processLogs) (110935) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:59,179::__init__::232::IOProcess::(_processLogs) (110935) Queuing response >Thread-81::DEBUG::2014-09-25 18:15:59,181::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:15:59,189::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000485537 s, 764 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:15:59,191::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:15:59,191::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:15:59,191::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:15:59,191::__init__::232::IOProcess::(_processLogs) (110936) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:15:59,192::__init__::232::IOProcess::(_processLogs) (110936) Queuing response >Dummy-148::DEBUG::2014-09-25 18:16:00,590::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:00,613::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135231 s, 75.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:01,729::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:01,731::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72031::DEBUG::2014-09-25 18:16:01,733::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:02,207::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:02,208::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72032::DEBUG::2014-09-25 18:16:02,208::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72032::DEBUG::2014-09-25 18:16:02,210::task::595::Storage.TaskManager.Task::(_updateState) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::moving from state init -> state preparing >Thread-72032::INFO::2014-09-25 18:16:02,210::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72032::ERROR::2014-09-25 18:16:02,211::task::866::Storage.TaskManager.Task::(_setError) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72032::DEBUG::2014-09-25 18:16:02,211::task::885::Storage.TaskManager.Task::(_run) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::Task._run: d60ec839-edfb-4514-ae81-c185dffbe475 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72032::DEBUG::2014-09-25 18:16:02,211::task::1217::Storage.TaskManager.Task::(stop) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::stopping in state preparing (force False) >Thread-72032::DEBUG::2014-09-25 18:16:02,212::task::993::Storage.TaskManager.Task::(_decref) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::ref 1 aborting True >Thread-72032::INFO::2014-09-25 18:16:02,212::task::1171::Storage.TaskManager.Task::(prepare) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::aborting: Task is aborted: u'' - code 100 >Thread-72032::DEBUG::2014-09-25 18:16:02,212::task::1176::Storage.TaskManager.Task::(prepare) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::Prepare: aborted: >Thread-72032::DEBUG::2014-09-25 18:16:02,212::task::993::Storage.TaskManager.Task::(_decref) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::ref 0 aborting True >Thread-72032::DEBUG::2014-09-25 18:16:02,213::task::928::Storage.TaskManager.Task::(_doAbort) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::Task._doAbort: force False >Thread-72032::DEBUG::2014-09-25 18:16:02,213::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72032::DEBUG::2014-09-25 18:16:02,213::task::595::Storage.TaskManager.Task::(_updateState) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::moving from state preparing -> state aborting >Thread-72032::DEBUG::2014-09-25 18:16:02,213::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::_aborting: recover policy none >Thread-72032::DEBUG::2014-09-25 18:16:02,214::task::595::Storage.TaskManager.Task::(_updateState) Task=`d60ec839-edfb-4514-ae81-c185dffbe475`::moving from state aborting -> state failed >Thread-72032::DEBUG::2014-09-25 18:16:02,214::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72032::DEBUG::2014-09-25 18:16:02,214::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72032::ERROR::2014-09-25 18:16:02,214::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72032::DEBUG::2014-09-25 18:16:02,215::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:02,633::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:02,659::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149991 s, 68.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:16:04,679::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:04,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189555 s, 54.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:04,745::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:04,747::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72033::DEBUG::2014-09-25 18:16:04,749::task::595::Storage.TaskManager.Task::(_updateState) Task=`7b979ff7-e5ec-4195-9e21-e55af7c55397`::moving from state init -> state preparing >Thread-72033::INFO::2014-09-25 18:16:04,750::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72033::INFO::2014-09-25 18:16:04,750::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000385713', 'lastCheck': '9.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '46.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000485537', 'lastCheck': '5.6', 'valid': True}} >Thread-72033::DEBUG::2014-09-25 18:16:04,750::task::1191::Storage.TaskManager.Task::(prepare) Task=`7b979ff7-e5ec-4195-9e21-e55af7c55397`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000385713', 'lastCheck': '9.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '46.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000485537', 'lastCheck': '5.6', 'valid': True}} >Thread-72033::DEBUG::2014-09-25 18:16:04,751::task::595::Storage.TaskManager.Task::(_updateState) Task=`7b979ff7-e5ec-4195-9e21-e55af7c55397`::moving from state preparing -> state finished >Thread-72033::DEBUG::2014-09-25 18:16:04,751::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72033::DEBUG::2014-09-25 18:16:04,751::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72033::DEBUG::2014-09-25 18:16:04,751::task::993::Storage.TaskManager.Task::(_decref) Task=`7b979ff7-e5ec-4195-9e21-e55af7c55397`::ref 0 aborting False >Thread-72033::DEBUG::2014-09-25 18:16:04,754::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:04,770::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:04,771::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72034::DEBUG::2014-09-25 18:16:04,773::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-72001::DEBUG::2014-09-25 18:16:05,208::hsm::2441::Storage.HSM::(connectStorageServer) prefetch failed: {u'3573f707-d0c4-4dc9-b09f-dca9a3506a4f': <function findDomain at 0x14be758>, u'fc587b6a-bac3-4668-99de-7f6fd6acce7f': <function findDomain at 0x14be758>, u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': <function findDomain at 0x14be758>} >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2438, in connectStorageServer > doms = self.__prefetchDomains(domType, conObj) > File "/usr/share/vdsm/storage/hsm.py", line 2375, in __prefetchDomains > goop.glob.glob(os.path.join(lPath, uuidPatern))) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-72001::DEBUG::2014-09-25 18:16:05,209::hsm::2452::Storage.HSM::(connectStorageServer) knownSDs: {3573f707-d0c4-4dc9-b09f-dca9a3506a4f: storage.nfsSD.findDomain, fc587b6a-bac3-4668-99de-7f6fd6acce7f: storage.nfsSD.findDomain, eb633636-3019-47a9-9f5a-fc81f8dfd7dc: storage.nfsSD.findDomain} >Thread-72001::INFO::2014-09-25 18:16:05,209::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8'}]} >Thread-72001::DEBUG::2014-09-25 18:16:05,210::task::1191::Storage.TaskManager.Task::(prepare) Task=`ecd90552-46d7-4a12-84cb-d10cac117193`::finished: {'statuslist': [{'status': 0, 'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8'}]} >Thread-72001::DEBUG::2014-09-25 18:16:05,210::task::595::Storage.TaskManager.Task::(_updateState) Task=`ecd90552-46d7-4a12-84cb-d10cac117193`::moving from state preparing -> state finished >Thread-72001::DEBUG::2014-09-25 18:16:05,210::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72001::DEBUG::2014-09-25 18:16:05,210::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72001::DEBUG::2014-09-25 18:16:05,211::task::993::Storage.TaskManager.Task::(_decref) Task=`ecd90552-46d7-4a12-84cb-d10cac117193`::ref 0 aborting False >Thread-72001::DEBUG::2014-09-25 18:16:05,211::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.connectStorageServer' in bridge with [{'status': 0, 'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8'}] >Thread-72001::DEBUG::2014-09-25 18:16:05,211::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:16:05,785::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:05,785::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:05,786::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:05,786::__init__::232::IOProcess::(_processLogs) (110937) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:05,792::__init__::232::IOProcess::(_processLogs) (110937) Queuing response >Thread-79::DEBUG::2014-09-25 18:16:05,793::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:16:05,808::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00151241 s, 372 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:05,810::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:05,810::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:05,810::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:05,810::__init__::232::IOProcess::(_processLogs) (110938) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:05,815::__init__::232::IOProcess::(_processLogs) (110938) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:05,816::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:05,816::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:05,817::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:05,817::__init__::232::IOProcess::(_processLogs) (110939) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:05,818::__init__::232::IOProcess::(_processLogs) (110939) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:05,818::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:05,819::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:05,819::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:05,819::__init__::232::IOProcess::(_processLogs) (110940) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:05,820::__init__::232::IOProcess::(_processLogs) (110940) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:05,820::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:05,820::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:05,822::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:05,822::__init__::232::IOProcess::(_processLogs) (110941) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:05,822::__init__::232::IOProcess::(_processLogs) (110941) Queuing response >Dummy-148::DEBUG::2014-09-25 18:16:06,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:06,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137847 s, 74.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:07,833::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:07,834::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72035::DEBUG::2014-09-25 18:16:07,836::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:08,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:08,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168111 s, 60.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:09,195::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:09,195::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:09,195::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:09,195::__init__::232::IOProcess::(_processLogs) (110942) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:09,196::__init__::232::IOProcess::(_processLogs) (110942) Queuing response >Thread-81::DEBUG::2014-09-25 18:16:09,197::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:16:09,206::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000419198 s, 885 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:09,208::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:09,208::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:09,208::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:09,209::__init__::232::IOProcess::(_processLogs) (110943) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:09,209::__init__::232::IOProcess::(_processLogs) (110943) Queuing response >Dummy-148::DEBUG::2014-09-25 18:16:10,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:10,848::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:10,850::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72036::DEBUG::2014-09-25 18:16:10,854::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:10,860::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0197246 s, 51.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:12,243::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72037::DEBUG::2014-09-25 18:16:12,244::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:16:12,244::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72037::DEBUG::2014-09-25 18:16:12,246::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::moving from state init -> state preparing >Thread-72037::INFO::2014-09-25 18:16:12,246::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72037::ERROR::2014-09-25 18:16:12,247::task::866::Storage.TaskManager.Task::(_setError) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72037::DEBUG::2014-09-25 18:16:12,247::task::885::Storage.TaskManager.Task::(_run) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::Task._run: b6a13306-e4b4-49b6-8186-dcdce1bfe00c (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72037::DEBUG::2014-09-25 18:16:12,247::task::1217::Storage.TaskManager.Task::(stop) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::stopping in state preparing (force False) >Thread-72037::DEBUG::2014-09-25 18:16:12,248::task::993::Storage.TaskManager.Task::(_decref) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::ref 1 aborting True >Thread-72037::INFO::2014-09-25 18:16:12,248::task::1171::Storage.TaskManager.Task::(prepare) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::aborting: Task is aborted: u'' - code 100 >Thread-72037::DEBUG::2014-09-25 18:16:12,248::task::1176::Storage.TaskManager.Task::(prepare) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::Prepare: aborted: >Thread-72037::DEBUG::2014-09-25 18:16:12,248::task::993::Storage.TaskManager.Task::(_decref) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::ref 0 aborting True >Thread-72037::DEBUG::2014-09-25 18:16:12,249::task::928::Storage.TaskManager.Task::(_doAbort) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::Task._doAbort: force False >Thread-72037::DEBUG::2014-09-25 18:16:12,249::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72037::DEBUG::2014-09-25 18:16:12,249::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::moving from state preparing -> state aborting >Thread-72037::DEBUG::2014-09-25 18:16:12,249::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::_aborting: recover policy none >Thread-72037::DEBUG::2014-09-25 18:16:12,249::task::595::Storage.TaskManager.Task::(_updateState) Task=`b6a13306-e4b4-49b6-8186-dcdce1bfe00c`::moving from state aborting -> state failed >Thread-72037::DEBUG::2014-09-25 18:16:12,250::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72037::DEBUG::2014-09-25 18:16:12,250::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72037::ERROR::2014-09-25 18:16:12,250::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72037::DEBUG::2014-09-25 18:16:12,251::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:12,880::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:12,908::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174904 s, 58.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:13,870::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:13,871::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72038::DEBUG::2014-09-25 18:16:13,873::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:14,928::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:14,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170657 s, 60.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:15,822::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:15,823::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:15,823::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:15,823::__init__::232::IOProcess::(_processLogs) (110944) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:15,826::__init__::232::IOProcess::(_processLogs) (110944) Queuing response >Thread-79::DEBUG::2014-09-25 18:16:15,827::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:16:15,844::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00238642 s, 235 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:15,845::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:15,846::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:15,846::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:15,846::__init__::232::IOProcess::(_processLogs) (110945) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:15,847::__init__::232::IOProcess::(_processLogs) (110945) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:15,849::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:15,850::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:15,850::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:15,850::__init__::232::IOProcess::(_processLogs) (110946) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:15,851::__init__::232::IOProcess::(_processLogs) (110946) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:15,851::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:15,852::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:15,852::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:15,853::__init__::232::IOProcess::(_processLogs) (110947) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:15,853::__init__::232::IOProcess::(_processLogs) (110947) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:15,853::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:15,854::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:15,855::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:15,855::__init__::232::IOProcess::(_processLogs) (110948) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:15,855::__init__::232::IOProcess::(_processLogs) (110948) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:16,889::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:16,892::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72039::DEBUG::2014-09-25 18:16:16,895::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:16,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:17,011::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181082 s, 56.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:16:19,026::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:19,060::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0203154 s, 50.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:19,211::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:19,212::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:19,212::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:19,212::__init__::232::IOProcess::(_processLogs) (110949) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:19,219::__init__::232::IOProcess::(_processLogs) (110949) Queuing response >Thread-81::DEBUG::2014-09-25 18:16:19,220::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:16:19,233::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0045023 s, 82.4 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:19,234::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:19,234::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:19,234::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:19,235::__init__::232::IOProcess::(_processLogs) (110950) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:19,239::__init__::232::IOProcess::(_processLogs) (110950) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:19,906::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:19,907::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72040::DEBUG::2014-09-25 18:16:19,910::task::595::Storage.TaskManager.Task::(_updateState) Task=`ec58b232-926e-41ff-aefc-69f066ffd7f1`::moving from state init -> state preparing >Thread-72040::INFO::2014-09-25 18:16:19,910::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72040::INFO::2014-09-25 18:16:19,910::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00238642', 'lastCheck': '4.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0045023', 'lastCheck': '0.7', 'valid': True}} >Thread-72040::DEBUG::2014-09-25 18:16:19,911::task::1191::Storage.TaskManager.Task::(prepare) Task=`ec58b232-926e-41ff-aefc-69f066ffd7f1`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00238642', 'lastCheck': '4.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '61.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0045023', 'lastCheck': '0.7', 'valid': True}} >Thread-72040::DEBUG::2014-09-25 18:16:19,911::task::595::Storage.TaskManager.Task::(_updateState) Task=`ec58b232-926e-41ff-aefc-69f066ffd7f1`::moving from state preparing -> state finished >Thread-72040::DEBUG::2014-09-25 18:16:19,911::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72040::DEBUG::2014-09-25 18:16:19,911::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72040::DEBUG::2014-09-25 18:16:19,911::task::993::Storage.TaskManager.Task::(_decref) Task=`ec58b232-926e-41ff-aefc-69f066ffd7f1`::ref 0 aborting False >Thread-72040::DEBUG::2014-09-25 18:16:19,914::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:19,939::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:19,940::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72041::DEBUG::2014-09-25 18:16:19,941::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:21,080::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:21,109::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0148573 s, 68.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:22,279::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72042::DEBUG::2014-09-25 18:16:22,280::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:16:22,280::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72042::DEBUG::2014-09-25 18:16:22,283::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::moving from state init -> state preparing >Thread-72042::INFO::2014-09-25 18:16:22,283::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72042::ERROR::2014-09-25 18:16:22,283::task::866::Storage.TaskManager.Task::(_setError) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72042::DEBUG::2014-09-25 18:16:22,284::task::885::Storage.TaskManager.Task::(_run) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::Task._run: 1a51db4f-7e15-47d5-b32b-bdc82a7d0aad (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72042::DEBUG::2014-09-25 18:16:22,284::task::1217::Storage.TaskManager.Task::(stop) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::stopping in state preparing (force False) >Thread-72042::DEBUG::2014-09-25 18:16:22,284::task::993::Storage.TaskManager.Task::(_decref) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::ref 1 aborting True >Thread-72042::INFO::2014-09-25 18:16:22,284::task::1171::Storage.TaskManager.Task::(prepare) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::aborting: Task is aborted: u'' - code 100 >Thread-72042::DEBUG::2014-09-25 18:16:22,285::task::1176::Storage.TaskManager.Task::(prepare) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::Prepare: aborted: >Thread-72042::DEBUG::2014-09-25 18:16:22,285::task::993::Storage.TaskManager.Task::(_decref) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::ref 0 aborting True >Thread-72042::DEBUG::2014-09-25 18:16:22,285::task::928::Storage.TaskManager.Task::(_doAbort) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::Task._doAbort: force False >Thread-72042::DEBUG::2014-09-25 18:16:22,285::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72042::DEBUG::2014-09-25 18:16:22,286::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::moving from state preparing -> state aborting >Thread-72042::DEBUG::2014-09-25 18:16:22,286::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::_aborting: recover policy none >Thread-72042::DEBUG::2014-09-25 18:16:22,286::task::595::Storage.TaskManager.Task::(_updateState) Task=`1a51db4f-7e15-47d5-b32b-bdc82a7d0aad`::moving from state aborting -> state failed >Thread-72042::DEBUG::2014-09-25 18:16:22,286::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72042::DEBUG::2014-09-25 18:16:22,286::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72042::ERROR::2014-09-25 18:16:22,287::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72042::DEBUG::2014-09-25 18:16:22,287::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:23,005::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:23,006::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72043::DEBUG::2014-09-25 18:16:23,008::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:23,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:23,162::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0221701 s, 46.2 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:16:25,182::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:25,209::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144753 s, 70.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:25,856::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:25,856::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:25,856::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:25,856::__init__::232::IOProcess::(_processLogs) (110951) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:25,877::__init__::232::IOProcess::(_processLogs) (110951) Queuing response >Thread-79::DEBUG::2014-09-25 18:16:25,878::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:16:25,903::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00890224 s, 63.1 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:25,904::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:25,904::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:25,905::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:25,905::__init__::232::IOProcess::(_processLogs) (110952) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:25,911::__init__::232::IOProcess::(_processLogs) (110952) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:25,913::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:25,913::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:25,914::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:25,914::__init__::232::IOProcess::(_processLogs) (110953) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:25,914::__init__::232::IOProcess::(_processLogs) (110953) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:25,915::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:25,915::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:25,916::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:25,916::__init__::232::IOProcess::(_processLogs) (110954) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:25,916::__init__::232::IOProcess::(_processLogs) (110954) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:25,917::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:25,917::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:25,918::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:25,919::__init__::232::IOProcess::(_processLogs) (110955) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:25,919::__init__::232::IOProcess::(_processLogs) (110955) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:26,021::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:26,022::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72044::DEBUG::2014-09-25 18:16:26,024::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:27,229::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:27,257::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170077 s, 60.2 MB/s\n'; <rc> = 0 >Thread-72013::WARNING::2014-09-25 18:16:28,036::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:16:28,037::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:16:28,037::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:29,038::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:29,040::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72045::DEBUG::2014-09-25 18:16:29,041::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:16:29,241::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:29,242::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:29,242::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:29,242::__init__::232::IOProcess::(_processLogs) (110956) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:29,245::__init__::232::IOProcess::(_processLogs) (110956) Queuing response >Thread-81::DEBUG::2014-09-25 18:16:29,245::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:16:29,264::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00546371 s, 67.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:29,265::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:29,266::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:29,266::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:29,266::__init__::232::IOProcess::(_processLogs) (110957) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:29,269::__init__::232::IOProcess::(_processLogs) (110957) Queuing response >Dummy-148::DEBUG::2014-09-25 18:16:29,276::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:29,302::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0145977 s, 70.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:16:31,334::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:31,364::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166135 s, 61.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:32,055::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:32,057::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72046::DEBUG::2014-09-25 18:16:32,059::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:32,313::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:32,315::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72047::DEBUG::2014-09-25 18:16:32,315::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72047::DEBUG::2014-09-25 18:16:32,316::task::595::Storage.TaskManager.Task::(_updateState) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::moving from state init -> state preparing >Thread-72047::INFO::2014-09-25 18:16:32,317::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72047::ERROR::2014-09-25 18:16:32,317::task::866::Storage.TaskManager.Task::(_setError) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72047::DEBUG::2014-09-25 18:16:32,318::task::885::Storage.TaskManager.Task::(_run) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::Task._run: e662fe9b-e6fb-4b84-a1db-eff71aa5e138 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72047::DEBUG::2014-09-25 18:16:32,318::task::1217::Storage.TaskManager.Task::(stop) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::stopping in state preparing (force False) >Thread-72047::DEBUG::2014-09-25 18:16:32,318::task::993::Storage.TaskManager.Task::(_decref) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::ref 1 aborting True >Thread-72047::INFO::2014-09-25 18:16:32,318::task::1171::Storage.TaskManager.Task::(prepare) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::aborting: Task is aborted: u'' - code 100 >Thread-72047::DEBUG::2014-09-25 18:16:32,318::task::1176::Storage.TaskManager.Task::(prepare) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::Prepare: aborted: >Thread-72047::DEBUG::2014-09-25 18:16:32,319::task::993::Storage.TaskManager.Task::(_decref) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::ref 0 aborting True >Thread-72047::DEBUG::2014-09-25 18:16:32,319::task::928::Storage.TaskManager.Task::(_doAbort) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::Task._doAbort: force False >Thread-72047::DEBUG::2014-09-25 18:16:32,319::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72047::DEBUG::2014-09-25 18:16:32,319::task::595::Storage.TaskManager.Task::(_updateState) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::moving from state preparing -> state aborting >Thread-72047::DEBUG::2014-09-25 18:16:32,320::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::_aborting: recover policy none >Thread-72047::DEBUG::2014-09-25 18:16:32,320::task::595::Storage.TaskManager.Task::(_updateState) Task=`e662fe9b-e6fb-4b84-a1db-eff71aa5e138`::moving from state aborting -> state failed >Thread-72047::DEBUG::2014-09-25 18:16:32,320::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72047::DEBUG::2014-09-25 18:16:32,320::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72047::ERROR::2014-09-25 18:16:32,320::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72047::DEBUG::2014-09-25 18:16:32,321::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:33,384::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:33,407::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134516 s, 76.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:35,071::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:35,072::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72048::DEBUG::2014-09-25 18:16:35,075::task::595::Storage.TaskManager.Task::(_updateState) Task=`ba631a80-4f2d-4058-8e45-1130c750df7c`::moving from state init -> state preparing >Thread-72048::INFO::2014-09-25 18:16:35,077::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72048::INFO::2014-09-25 18:16:35,083::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00890224', 'lastCheck': '9.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00546371', 'lastCheck': '5.8', 'valid': True}} >Thread-72048::DEBUG::2014-09-25 18:16:35,083::task::1191::Storage.TaskManager.Task::(prepare) Task=`ba631a80-4f2d-4058-8e45-1130c750df7c`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00890224', 'lastCheck': '9.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '7.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00546371', 'lastCheck': '5.8', 'valid': True}} >Thread-72048::DEBUG::2014-09-25 18:16:35,084::task::595::Storage.TaskManager.Task::(_updateState) Task=`ba631a80-4f2d-4058-8e45-1130c750df7c`::moving from state preparing -> state finished >Thread-72048::DEBUG::2014-09-25 18:16:35,084::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72048::DEBUG::2014-09-25 18:16:35,084::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72048::DEBUG::2014-09-25 18:16:35,084::task::993::Storage.TaskManager.Task::(_decref) Task=`ba631a80-4f2d-4058-8e45-1130c750df7c`::ref 0 aborting False >Thread-72048::DEBUG::2014-09-25 18:16:35,088::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:35,104::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:35,105::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72049::DEBUG::2014-09-25 18:16:35,107::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:35,426::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:35,466::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169474 s, 60.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:35,919::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:35,920::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:35,920::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:35,920::__init__::232::IOProcess::(_processLogs) (110958) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:35,923::__init__::232::IOProcess::(_processLogs) (110958) Queuing response >Thread-79::DEBUG::2014-09-25 18:16:35,924::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:16:35,941::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00210719 s, 267 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:35,942::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:35,942::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:35,942::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:35,943::__init__::232::IOProcess::(_processLogs) (110959) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:35,948::__init__::232::IOProcess::(_processLogs) (110959) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:35,949::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:35,949::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:35,949::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:35,950::__init__::232::IOProcess::(_processLogs) (110960) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:35,950::__init__::232::IOProcess::(_processLogs) (110960) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:35,951::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:35,951::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:35,952::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:35,952::__init__::232::IOProcess::(_processLogs) (110961) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:35,952::__init__::232::IOProcess::(_processLogs) (110961) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:35,953::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:35,953::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:35,953::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:35,954::__init__::232::IOProcess::(_processLogs) (110962) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:35,954::__init__::232::IOProcess::(_processLogs) (110962) Queuing response >Dummy-148::DEBUG::2014-09-25 18:16:37,486::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:37,517::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0183003 s, 56.0 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 18:16:38,038::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain fc587b6a-bac3-4668-99de-7f6fd6acce7f >Thread-80::DEBUG::2014-09-25 18:16:38,039::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-80::DEBUG::2014-09-25 18:16:38,039::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-80::DEBUG::2014-09-25 18:16:38,039::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-80::DEBUG::2014-09-25 18:16:38,039::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-80::DEBUG::2014-09-25 18:16:38,040::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-80::DEBUG::2014-09-25 18:16:38,040::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) >Thread-80::DEBUG::2014-09-25 18:16:38,063::misc::751::Storage.SamplingMethod::(__call__) Returning last result >Thread-80::DEBUG::2014-09-25 18:16:38,066::multipath::110::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /sbin/multipath (cwd None) >Thread-80::DEBUG::2014-09-25 18:16:38,144::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-80::DEBUG::2014-09-25 18:16:38,146::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:38,147::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-80::DEBUG::2014-09-25 18:16:38,148::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >JsonRpcServer::DEBUG::2014-09-25 18:16:38,150::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-80::DEBUG::2014-09-25 18:16:38,151::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-80::DEBUG::2014-09-25 18:16:38,153::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-80::DEBUG::2014-09-25 18:16:38,155::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-72050::DEBUG::2014-09-25 18:16:38,157::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-80::DEBUG::2014-09-25 18:16:38,157::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-80::DEBUG::2014-09-25 18:16:38,159::misc::751::Storage.SamplingMethod::(__call__) Returning last result >Thread-29::DEBUG::2014-09-25 18:16:38,164::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:38,164::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:38,165::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:38,165::__init__::232::IOProcess::(_processLogs) (110963) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:16:38,165::__init__::232::IOProcess::(_processLogs) (110963) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:38,166::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:38,167::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:38,167::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:38,167::__init__::232::IOProcess::(_processLogs) (110964) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:16:38,167::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:38,168::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:38,168::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:38,169::__init__::232::IOProcess::(_processLogs) (110965) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:16:38,169::__init__::232::IOProcess::(_processLogs) (110965) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:39,271::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:39,271::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:39,271::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:39,271::__init__::232::IOProcess::(_processLogs) (110966) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:39,273::__init__::232::IOProcess::(_processLogs) (110966) Queuing response >Thread-81::DEBUG::2014-09-25 18:16:39,274::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:16:39,290::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00184682 s, 201 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:39,291::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:39,291::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:39,292::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:39,292::__init__::232::IOProcess::(_processLogs) (110967) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:39,292::__init__::232::IOProcess::(_processLogs) (110967) Queuing response >Dummy-148::DEBUG::2014-09-25 18:16:39,537::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:39,563::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151163 s, 67.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:41,171::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:41,172::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72054::DEBUG::2014-09-25 18:16:41,174::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:41,583::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:41,614::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172764 s, 59.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:42,341::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72055::DEBUG::2014-09-25 18:16:42,342::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:16:42,342::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72055::DEBUG::2014-09-25 18:16:42,344::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::moving from state init -> state preparing >Thread-72055::INFO::2014-09-25 18:16:42,345::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72055::ERROR::2014-09-25 18:16:42,345::task::866::Storage.TaskManager.Task::(_setError) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72055::DEBUG::2014-09-25 18:16:42,346::task::885::Storage.TaskManager.Task::(_run) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::Task._run: 0c1e120e-6c66-4a94-86f4-12b077f37d22 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72055::DEBUG::2014-09-25 18:16:42,346::task::1217::Storage.TaskManager.Task::(stop) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::stopping in state preparing (force False) >Thread-72055::DEBUG::2014-09-25 18:16:42,346::task::993::Storage.TaskManager.Task::(_decref) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::ref 1 aborting True >Thread-72055::INFO::2014-09-25 18:16:42,346::task::1171::Storage.TaskManager.Task::(prepare) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::aborting: Task is aborted: u'' - code 100 >Thread-72055::DEBUG::2014-09-25 18:16:42,347::task::1176::Storage.TaskManager.Task::(prepare) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::Prepare: aborted: >Thread-72055::DEBUG::2014-09-25 18:16:42,347::task::993::Storage.TaskManager.Task::(_decref) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::ref 0 aborting True >Thread-72055::DEBUG::2014-09-25 18:16:42,347::task::928::Storage.TaskManager.Task::(_doAbort) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::Task._doAbort: force False >Thread-72055::DEBUG::2014-09-25 18:16:42,347::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72055::DEBUG::2014-09-25 18:16:42,348::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::moving from state preparing -> state aborting >Thread-72055::DEBUG::2014-09-25 18:16:42,348::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::_aborting: recover policy none >Thread-72055::DEBUG::2014-09-25 18:16:42,348::task::595::Storage.TaskManager.Task::(_updateState) Task=`0c1e120e-6c66-4a94-86f4-12b077f37d22`::moving from state aborting -> state failed >Thread-72055::DEBUG::2014-09-25 18:16:42,348::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72055::DEBUG::2014-09-25 18:16:42,348::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72055::ERROR::2014-09-25 18:16:42,349::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72055::DEBUG::2014-09-25 18:16:42,349::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:43,634::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:43,664::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147229 s, 69.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:44,187::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:44,188::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72056::DEBUG::2014-09-25 18:16:44,190::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:45,684::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:45,711::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0149535 s, 68.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:45,955::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:45,956::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:45,956::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:45,956::__init__::232::IOProcess::(_processLogs) (110968) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:45,959::__init__::232::IOProcess::(_processLogs) (110968) Queuing response >Thread-79::DEBUG::2014-09-25 18:16:45,960::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:16:45,995::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00460653 s, 122 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:45,996::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:45,996::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:45,997::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:45,997::__init__::232::IOProcess::(_processLogs) (110969) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:46,000::__init__::232::IOProcess::(_processLogs) (110969) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:46,002::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:46,003::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:46,003::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:46,004::__init__::232::IOProcess::(_processLogs) (110970) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:46,004::__init__::232::IOProcess::(_processLogs) (110970) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:46,004::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:46,005::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:46,005::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:46,006::__init__::232::IOProcess::(_processLogs) (110971) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:46,006::__init__::232::IOProcess::(_processLogs) (110971) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:46,007::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:46,007::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:46,008::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:46,009::__init__::232::IOProcess::(_processLogs) (110972) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:46,009::__init__::232::IOProcess::(_processLogs) (110972) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:47,203::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:47,204::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72057::DEBUG::2014-09-25 18:16:47,206::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:47,732::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:47,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161103 s, 63.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:49,294::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:49,295::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:49,295::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:49,295::__init__::232::IOProcess::(_processLogs) (110973) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:49,295::__init__::232::IOProcess::(_processLogs) (110973) Queuing response >Thread-81::DEBUG::2014-09-25 18:16:49,297::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:16:49,323::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00353157 s, 105 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:49,324::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:49,325::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:49,325::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:49,325::__init__::232::IOProcess::(_processLogs) (110974) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:49,331::__init__::232::IOProcess::(_processLogs) (110974) Queuing response >Dummy-148::DEBUG::2014-09-25 18:16:49,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:49,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136901 s, 74.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:50,219::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:50,220::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72058::DEBUG::2014-09-25 18:16:50,223::task::595::Storage.TaskManager.Task::(_updateState) Task=`4c2c475a-47f8-4e92-aa1d-03de138e9d85`::moving from state init -> state preparing >Thread-72058::INFO::2014-09-25 18:16:50,223::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72058::INFO::2014-09-25 18:16:50,224::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00460653', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '22.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00353157', 'lastCheck': '0.9', 'valid': True}} >Thread-72058::DEBUG::2014-09-25 18:16:50,224::task::1191::Storage.TaskManager.Task::(prepare) Task=`4c2c475a-47f8-4e92-aa1d-03de138e9d85`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00460653', 'lastCheck': '4.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '22.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00353157', 'lastCheck': '0.9', 'valid': True}} >Thread-72058::DEBUG::2014-09-25 18:16:50,224::task::595::Storage.TaskManager.Task::(_updateState) Task=`4c2c475a-47f8-4e92-aa1d-03de138e9d85`::moving from state preparing -> state finished >Thread-72058::DEBUG::2014-09-25 18:16:50,224::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72058::DEBUG::2014-09-25 18:16:50,225::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72058::DEBUG::2014-09-25 18:16:50,225::task::993::Storage.TaskManager.Task::(_decref) Task=`4c2c475a-47f8-4e92-aa1d-03de138e9d85`::ref 0 aborting False >Thread-72058::DEBUG::2014-09-25 18:16:50,227::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:50,257::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:50,258::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72059::DEBUG::2014-09-25 18:16:50,260::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:51,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:51,851::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.016107 s, 63.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:52,373::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72060::DEBUG::2014-09-25 18:16:52,374::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:16:52,374::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72060::DEBUG::2014-09-25 18:16:52,377::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::moving from state init -> state preparing >Thread-72060::INFO::2014-09-25 18:16:52,377::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72060::ERROR::2014-09-25 18:16:52,377::task::866::Storage.TaskManager.Task::(_setError) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72060::DEBUG::2014-09-25 18:16:52,378::task::885::Storage.TaskManager.Task::(_run) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::Task._run: fc43c367-2e80-4004-99f6-828aee26b8df (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72060::DEBUG::2014-09-25 18:16:52,378::task::1217::Storage.TaskManager.Task::(stop) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::stopping in state preparing (force False) >Thread-72060::DEBUG::2014-09-25 18:16:52,378::task::993::Storage.TaskManager.Task::(_decref) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::ref 1 aborting True >Thread-72060::INFO::2014-09-25 18:16:52,379::task::1171::Storage.TaskManager.Task::(prepare) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::aborting: Task is aborted: u'' - code 100 >Thread-72060::DEBUG::2014-09-25 18:16:52,379::task::1176::Storage.TaskManager.Task::(prepare) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::Prepare: aborted: >Thread-72060::DEBUG::2014-09-25 18:16:52,379::task::993::Storage.TaskManager.Task::(_decref) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::ref 0 aborting True >Thread-72060::DEBUG::2014-09-25 18:16:52,379::task::928::Storage.TaskManager.Task::(_doAbort) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::Task._doAbort: force False >Thread-72060::DEBUG::2014-09-25 18:16:52,379::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72060::DEBUG::2014-09-25 18:16:52,380::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::moving from state preparing -> state aborting >Thread-72060::DEBUG::2014-09-25 18:16:52,380::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::_aborting: recover policy none >Thread-72060::DEBUG::2014-09-25 18:16:52,380::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc43c367-2e80-4004-99f6-828aee26b8df`::moving from state aborting -> state failed >Thread-72060::DEBUG::2014-09-25 18:16:52,380::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72060::DEBUG::2014-09-25 18:16:52,380::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72060::ERROR::2014-09-25 18:16:52,381::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72060::DEBUG::2014-09-25 18:16:52,381::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:53,301::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:53,303::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72061::DEBUG::2014-09-25 18:16:53,304::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:53,871::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:53,895::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136497 s, 75.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:16:55,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:55,947::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.021426 s, 47.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:56,009::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:56,009::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:56,010::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:56,010::__init__::232::IOProcess::(_processLogs) (110975) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:56,011::__init__::232::IOProcess::(_processLogs) (110975) Queuing response >Thread-79::DEBUG::2014-09-25 18:16:56,012::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:16:56,020::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000281125 s, 2.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:56,021::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:56,022::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:56,022::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:56,023::__init__::232::IOProcess::(_processLogs) (110976) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:56,024::__init__::232::IOProcess::(_processLogs) (110976) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:56,024::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:56,024::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:56,025::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:56,025::__init__::232::IOProcess::(_processLogs) (110977) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:56,026::__init__::232::IOProcess::(_processLogs) (110977) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:56,026::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:56,026::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:56,027::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:56,027::__init__::232::IOProcess::(_processLogs) (110978) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:56,028::__init__::232::IOProcess::(_processLogs) (110978) Queuing response >Thread-29::DEBUG::2014-09-25 18:16:56,028::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:56,028::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:56,029::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:56,029::__init__::232::IOProcess::(_processLogs) (110979) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:16:56,029::__init__::232::IOProcess::(_processLogs) (110979) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:56,324::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:56,325::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72062::DEBUG::2014-09-25 18:16:56,327::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:16:57,966::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:16:57,994::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179609 s, 57.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:59,333::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:59,333::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:59,333::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:59,334::__init__::232::IOProcess::(_processLogs) (110980) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:59,336::__init__::232::IOProcess::(_processLogs) (110980) Queuing response >Thread-81::DEBUG::2014-09-25 18:16:59,337::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:16:59,345::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:16:59,347::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72063::DEBUG::2014-09-25 18:16:59,349::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-81::DEBUG::2014-09-25 18:16:59,352::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00298758 s, 124 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:16:59,353::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:16:59,353::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:16:59,354::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:16:59,354::__init__::232::IOProcess::(_processLogs) (110981) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:16:59,357::__init__::232::IOProcess::(_processLogs) (110981) Queuing response >Dummy-148::DEBUG::2014-09-25 18:17:00,013::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:00,042::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0165841 s, 61.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:17:02,065::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:02,094::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174401 s, 58.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:02,362::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:02,364::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72064::DEBUG::2014-09-25 18:17:02,366::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:02,415::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72065::DEBUG::2014-09-25 18:17:02,416::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:17:02,416::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72065::DEBUG::2014-09-25 18:17:02,419::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::moving from state init -> state preparing >Thread-72065::INFO::2014-09-25 18:17:02,419::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72065::ERROR::2014-09-25 18:17:02,419::task::866::Storage.TaskManager.Task::(_setError) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72065::DEBUG::2014-09-25 18:17:02,420::task::885::Storage.TaskManager.Task::(_run) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::Task._run: 8d3d3aa7-85fa-4d46-a430-838cbad02578 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72065::DEBUG::2014-09-25 18:17:02,420::task::1217::Storage.TaskManager.Task::(stop) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::stopping in state preparing (force False) >Thread-72065::DEBUG::2014-09-25 18:17:02,420::task::993::Storage.TaskManager.Task::(_decref) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::ref 1 aborting True >Thread-72065::INFO::2014-09-25 18:17:02,421::task::1171::Storage.TaskManager.Task::(prepare) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::aborting: Task is aborted: u'' - code 100 >Thread-72065::DEBUG::2014-09-25 18:17:02,421::task::1176::Storage.TaskManager.Task::(prepare) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::Prepare: aborted: >Thread-72065::DEBUG::2014-09-25 18:17:02,421::task::993::Storage.TaskManager.Task::(_decref) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::ref 0 aborting True >Thread-72065::DEBUG::2014-09-25 18:17:02,421::task::928::Storage.TaskManager.Task::(_doAbort) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::Task._doAbort: force False >Thread-72065::DEBUG::2014-09-25 18:17:02,421::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72065::DEBUG::2014-09-25 18:17:02,422::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::moving from state preparing -> state aborting >Thread-72065::DEBUG::2014-09-25 18:17:02,422::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::_aborting: recover policy none >Thread-72065::DEBUG::2014-09-25 18:17:02,422::task::595::Storage.TaskManager.Task::(_updateState) Task=`8d3d3aa7-85fa-4d46-a430-838cbad02578`::moving from state aborting -> state failed >Thread-72065::DEBUG::2014-09-25 18:17:02,422::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72065::DEBUG::2014-09-25 18:17:02,423::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72065::ERROR::2014-09-25 18:17:02,423::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72065::DEBUG::2014-09-25 18:17:02,424::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:04,114::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:04,140::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155713 s, 65.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:05,378::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:05,379::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72066::DEBUG::2014-09-25 18:17:05,382::task::595::Storage.TaskManager.Task::(_updateState) Task=`db5c3645-2257-4f4c-b0cd-1af937ba1f15`::moving from state init -> state preparing >Thread-72066::INFO::2014-09-25 18:17:05,382::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72066::INFO::2014-09-25 18:17:05,383::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000281125', 'lastCheck': '9.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '37.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00298758', 'lastCheck': '6.0', 'valid': True}} >Thread-72066::DEBUG::2014-09-25 18:17:05,383::task::1191::Storage.TaskManager.Task::(prepare) Task=`db5c3645-2257-4f4c-b0cd-1af937ba1f15`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000281125', 'lastCheck': '9.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '37.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00298758', 'lastCheck': '6.0', 'valid': True}} >Thread-72066::DEBUG::2014-09-25 18:17:05,383::task::595::Storage.TaskManager.Task::(_updateState) Task=`db5c3645-2257-4f4c-b0cd-1af937ba1f15`::moving from state preparing -> state finished >Thread-72066::DEBUG::2014-09-25 18:17:05,384::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72066::DEBUG::2014-09-25 18:17:05,384::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72066::DEBUG::2014-09-25 18:17:05,384::task::993::Storage.TaskManager.Task::(_decref) Task=`db5c3645-2257-4f4c-b0cd-1af937ba1f15`::ref 0 aborting False >Thread-72066::DEBUG::2014-09-25 18:17:05,386::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:05,405::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:05,406::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72067::DEBUG::2014-09-25 18:17:05,408::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:17:06,031::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:06,031::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:06,031::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:06,032::__init__::232::IOProcess::(_processLogs) (110982) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:06,032::__init__::232::IOProcess::(_processLogs) (110982) Queuing response >Thread-79::DEBUG::2014-09-25 18:17:06,033::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:17:06,053::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00287073 s, 196 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:06,054::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:06,054::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:06,054::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:06,055::__init__::232::IOProcess::(_processLogs) (110983) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:06,060::__init__::232::IOProcess::(_processLogs) (110983) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:06,062::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:06,062::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:06,063::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:06,063::__init__::232::IOProcess::(_processLogs) (110984) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:06,063::__init__::232::IOProcess::(_processLogs) (110984) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:06,064::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:06,064::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:06,065::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:06,065::__init__::232::IOProcess::(_processLogs) (110985) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:06,066::__init__::232::IOProcess::(_processLogs) (110985) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:06,066::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:06,066::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:06,067::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:06,068::__init__::232::IOProcess::(_processLogs) (110986) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:06,068::__init__::232::IOProcess::(_processLogs) (110986) Queuing response >Dummy-148::DEBUG::2014-09-25 18:17:06,160::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:06,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215491 s, 47.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:17:08,216::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:08,242::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138508 s, 73.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:08,452::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:08,453::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72068::DEBUG::2014-09-25 18:17:08,455::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:17:09,359::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:09,359::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:09,360::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:09,360::__init__::232::IOProcess::(_processLogs) (110987) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:09,369::__init__::232::IOProcess::(_processLogs) (110987) Queuing response >Thread-81::DEBUG::2014-09-25 18:17:09,370::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:17:09,406::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00405026 s, 91.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:09,407::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:09,407::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:09,407::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:09,408::__init__::232::IOProcess::(_processLogs) (110988) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:09,412::__init__::232::IOProcess::(_processLogs) (110988) Queuing response >Dummy-148::DEBUG::2014-09-25 18:17:10,262::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:10,291::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142416 s, 71.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:11,470::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:11,474::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72069::DEBUG::2014-09-25 18:17:11,479::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:12,314::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:12,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0137403 s, 74.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:12,449::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72070::DEBUG::2014-09-25 18:17:12,450::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:17:12,450::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72070::DEBUG::2014-09-25 18:17:12,453::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::moving from state init -> state preparing >Thread-72070::INFO::2014-09-25 18:17:12,453::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72070::ERROR::2014-09-25 18:17:12,454::task::866::Storage.TaskManager.Task::(_setError) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72070::DEBUG::2014-09-25 18:17:12,454::task::885::Storage.TaskManager.Task::(_run) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::Task._run: 5d43f1f2-89e1-4ea7-9776-4805a33f22c7 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72070::DEBUG::2014-09-25 18:17:12,454::task::1217::Storage.TaskManager.Task::(stop) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::stopping in state preparing (force False) >Thread-72070::DEBUG::2014-09-25 18:17:12,455::task::993::Storage.TaskManager.Task::(_decref) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::ref 1 aborting True >Thread-72070::INFO::2014-09-25 18:17:12,455::task::1171::Storage.TaskManager.Task::(prepare) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::aborting: Task is aborted: u'' - code 100 >Thread-72070::DEBUG::2014-09-25 18:17:12,455::task::1176::Storage.TaskManager.Task::(prepare) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::Prepare: aborted: >Thread-72070::DEBUG::2014-09-25 18:17:12,455::task::993::Storage.TaskManager.Task::(_decref) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::ref 0 aborting True >Thread-72070::DEBUG::2014-09-25 18:17:12,456::task::928::Storage.TaskManager.Task::(_doAbort) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::Task._doAbort: force False >Thread-72070::DEBUG::2014-09-25 18:17:12,456::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72070::DEBUG::2014-09-25 18:17:12,456::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::moving from state preparing -> state aborting >Thread-72070::DEBUG::2014-09-25 18:17:12,456::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::_aborting: recover policy none >Thread-72070::DEBUG::2014-09-25 18:17:12,456::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d43f1f2-89e1-4ea7-9776-4805a33f22c7`::moving from state aborting -> state failed >Thread-72070::DEBUG::2014-09-25 18:17:12,457::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72070::DEBUG::2014-09-25 18:17:12,457::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72070::ERROR::2014-09-25 18:17:12,457::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72070::DEBUG::2014-09-25 18:17:12,457::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:14,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:14,389::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171893 s, 59.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:14,496::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:14,497::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72071::DEBUG::2014-09-25 18:17:14,499::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:17:16,069::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:16,071::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:16,072::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:16,073::__init__::232::IOProcess::(_processLogs) (110989) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:16,075::__init__::232::IOProcess::(_processLogs) (110989) Queuing response >Thread-79::DEBUG::2014-09-25 18:17:16,077::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:17:16,101::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00927289 s, 60.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:16,104::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:16,105::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:16,106::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:16,107::__init__::232::IOProcess::(_processLogs) (110990) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:16,109::__init__::232::IOProcess::(_processLogs) (110990) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:16,112::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:16,113::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:16,115::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:16,116::__init__::232::IOProcess::(_processLogs) (110991) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:16,117::__init__::232::IOProcess::(_processLogs) (110991) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:16,118::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:16,119::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:16,121::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:16,122::__init__::232::IOProcess::(_processLogs) (110992) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:16,123::__init__::232::IOProcess::(_processLogs) (110992) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:16,125::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:16,126::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:16,127::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:16,128::__init__::232::IOProcess::(_processLogs) (110993) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:16,129::__init__::232::IOProcess::(_processLogs) (110993) Queuing response >Dummy-148::DEBUG::2014-09-25 18:17:16,410::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:16,455::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0279607 s, 36.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:17,513::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:17,515::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72072::DEBUG::2014-09-25 18:17:17,517::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:18,653::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:18,688::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199545 s, 51.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:19,414::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:19,415::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:19,415::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:19,415::__init__::232::IOProcess::(_processLogs) (110994) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:19,418::__init__::232::IOProcess::(_processLogs) (110994) Queuing response >Thread-81::DEBUG::2014-09-25 18:17:19,419::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:17:19,441::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00438213 s, 84.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:19,442::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:19,442::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:19,443::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:19,443::__init__::232::IOProcess::(_processLogs) (110995) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:19,449::__init__::232::IOProcess::(_processLogs) (110995) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:20,528::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:20,530::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72073::DEBUG::2014-09-25 18:17:20,533::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a8583da-bea0-43f0-bbaf-55f38a242fd8`::moving from state init -> state preparing >Thread-72073::INFO::2014-09-25 18:17:20,533::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72073::INFO::2014-09-25 18:17:20,533::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00927289', 'lastCheck': '4.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '52.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00438213', 'lastCheck': '1.1', 'valid': True}} >Thread-72073::DEBUG::2014-09-25 18:17:20,534::task::1191::Storage.TaskManager.Task::(prepare) Task=`3a8583da-bea0-43f0-bbaf-55f38a242fd8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00927289', 'lastCheck': '4.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '52.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00438213', 'lastCheck': '1.1', 'valid': True}} >Thread-72073::DEBUG::2014-09-25 18:17:20,534::task::595::Storage.TaskManager.Task::(_updateState) Task=`3a8583da-bea0-43f0-bbaf-55f38a242fd8`::moving from state preparing -> state finished >Thread-72073::DEBUG::2014-09-25 18:17:20,534::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72073::DEBUG::2014-09-25 18:17:20,534::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72073::DEBUG::2014-09-25 18:17:20,535::task::993::Storage.TaskManager.Task::(_decref) Task=`3a8583da-bea0-43f0-bbaf-55f38a242fd8`::ref 0 aborting False >Thread-72073::DEBUG::2014-09-25 18:17:20,537::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:20,559::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:20,560::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72074::DEBUG::2014-09-25 18:17:20,562::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:20,708::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:20,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162864 s, 62.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:22,482::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:22,483::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72075::DEBUG::2014-09-25 18:17:22,483::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72075::DEBUG::2014-09-25 18:17:22,485::task::595::Storage.TaskManager.Task::(_updateState) Task=`028ade66-5094-423c-9eca-00c18862bb51`::moving from state init -> state preparing >Thread-72075::INFO::2014-09-25 18:17:22,486::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72075::ERROR::2014-09-25 18:17:22,486::task::866::Storage.TaskManager.Task::(_setError) Task=`028ade66-5094-423c-9eca-00c18862bb51`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72075::DEBUG::2014-09-25 18:17:22,487::task::885::Storage.TaskManager.Task::(_run) Task=`028ade66-5094-423c-9eca-00c18862bb51`::Task._run: 028ade66-5094-423c-9eca-00c18862bb51 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72075::DEBUG::2014-09-25 18:17:22,487::task::1217::Storage.TaskManager.Task::(stop) Task=`028ade66-5094-423c-9eca-00c18862bb51`::stopping in state preparing (force False) >Thread-72075::DEBUG::2014-09-25 18:17:22,487::task::993::Storage.TaskManager.Task::(_decref) Task=`028ade66-5094-423c-9eca-00c18862bb51`::ref 1 aborting True >Thread-72075::INFO::2014-09-25 18:17:22,487::task::1171::Storage.TaskManager.Task::(prepare) Task=`028ade66-5094-423c-9eca-00c18862bb51`::aborting: Task is aborted: u'' - code 100 >Thread-72075::DEBUG::2014-09-25 18:17:22,488::task::1176::Storage.TaskManager.Task::(prepare) Task=`028ade66-5094-423c-9eca-00c18862bb51`::Prepare: aborted: >Thread-72075::DEBUG::2014-09-25 18:17:22,488::task::993::Storage.TaskManager.Task::(_decref) Task=`028ade66-5094-423c-9eca-00c18862bb51`::ref 0 aborting True >Thread-72075::DEBUG::2014-09-25 18:17:22,488::task::928::Storage.TaskManager.Task::(_doAbort) Task=`028ade66-5094-423c-9eca-00c18862bb51`::Task._doAbort: force False >Thread-72075::DEBUG::2014-09-25 18:17:22,488::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72075::DEBUG::2014-09-25 18:17:22,488::task::595::Storage.TaskManager.Task::(_updateState) Task=`028ade66-5094-423c-9eca-00c18862bb51`::moving from state preparing -> state aborting >Thread-72075::DEBUG::2014-09-25 18:17:22,489::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`028ade66-5094-423c-9eca-00c18862bb51`::_aborting: recover policy none >Thread-72075::DEBUG::2014-09-25 18:17:22,489::task::595::Storage.TaskManager.Task::(_updateState) Task=`028ade66-5094-423c-9eca-00c18862bb51`::moving from state aborting -> state failed >Thread-72075::DEBUG::2014-09-25 18:17:22,489::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72075::DEBUG::2014-09-25 18:17:22,489::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72075::ERROR::2014-09-25 18:17:22,489::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72075::DEBUG::2014-09-25 18:17:22,490::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:22,752::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:22,782::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171089 s, 59.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:23,611::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:23,612::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72076::DEBUG::2014-09-25 18:17:23,615::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:24,802::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:24,824::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134206 s, 76.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:26,132::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:26,133::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:26,133::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:26,133::__init__::232::IOProcess::(_processLogs) (110996) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:26,257::__init__::232::IOProcess::(_processLogs) (110996) Queuing response >Thread-79::DEBUG::2014-09-25 18:17:26,258::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:17:26,278::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00318308 s, 177 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:26,280::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:26,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:26,280::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:26,280::__init__::232::IOProcess::(_processLogs) (110997) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:26,285::__init__::232::IOProcess::(_processLogs) (110997) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:26,286::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:26,287::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:26,288::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:26,288::__init__::232::IOProcess::(_processLogs) (110998) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:26,288::__init__::232::IOProcess::(_processLogs) (110998) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:26,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:26,289::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:26,289::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:26,290::__init__::232::IOProcess::(_processLogs) (110999) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:26,290::__init__::232::IOProcess::(_processLogs) (110999) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:26,291::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:26,291::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:26,291::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:26,292::__init__::232::IOProcess::(_processLogs) (111000) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:26,292::__init__::232::IOProcess::(_processLogs) (111000) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:26,627::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:26,628::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72077::DEBUG::2014-09-25 18:17:26,631::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:26,844::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:26,873::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0153369 s, 66.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:17:28,893::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:28,924::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0208527 s, 49.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:29,451::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:29,452::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:29,452::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:29,452::__init__::232::IOProcess::(_processLogs) (111001) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:29,461::__init__::232::IOProcess::(_processLogs) (111001) Queuing response >Thread-81::DEBUG::2014-09-25 18:17:29,462::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:17:29,478::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00288537 s, 129 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:29,479::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:29,480::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:29,480::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:29,480::__init__::232::IOProcess::(_processLogs) (111002) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:29,483::__init__::232::IOProcess::(_processLogs) (111002) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:29,646::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:29,647::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72078::DEBUG::2014-09-25 18:17:29,649::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:30,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:30,975::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213397 s, 48.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:32,514::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72079::DEBUG::2014-09-25 18:17:32,515::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:17:32,515::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72079::DEBUG::2014-09-25 18:17:32,518::task::595::Storage.TaskManager.Task::(_updateState) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::moving from state init -> state preparing >Thread-72079::INFO::2014-09-25 18:17:32,518::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72079::ERROR::2014-09-25 18:17:32,518::task::866::Storage.TaskManager.Task::(_setError) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72079::DEBUG::2014-09-25 18:17:32,519::task::885::Storage.TaskManager.Task::(_run) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::Task._run: 176f565b-9595-4c79-82eb-05b8413c323e (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72079::DEBUG::2014-09-25 18:17:32,519::task::1217::Storage.TaskManager.Task::(stop) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::stopping in state preparing (force False) >Thread-72079::DEBUG::2014-09-25 18:17:32,519::task::993::Storage.TaskManager.Task::(_decref) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::ref 1 aborting True >Thread-72079::INFO::2014-09-25 18:17:32,520::task::1171::Storage.TaskManager.Task::(prepare) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::aborting: Task is aborted: u'' - code 100 >Thread-72079::DEBUG::2014-09-25 18:17:32,520::task::1176::Storage.TaskManager.Task::(prepare) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::Prepare: aborted: >Thread-72079::DEBUG::2014-09-25 18:17:32,520::task::993::Storage.TaskManager.Task::(_decref) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::ref 0 aborting True >Thread-72079::DEBUG::2014-09-25 18:17:32,520::task::928::Storage.TaskManager.Task::(_doAbort) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::Task._doAbort: force False >Thread-72079::DEBUG::2014-09-25 18:17:32,520::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72079::DEBUG::2014-09-25 18:17:32,521::task::595::Storage.TaskManager.Task::(_updateState) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::moving from state preparing -> state aborting >Thread-72079::DEBUG::2014-09-25 18:17:32,521::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::_aborting: recover policy none >Thread-72079::DEBUG::2014-09-25 18:17:32,521::task::595::Storage.TaskManager.Task::(_updateState) Task=`176f565b-9595-4c79-82eb-05b8413c323e`::moving from state aborting -> state failed >Thread-72079::DEBUG::2014-09-25 18:17:32,521::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72079::DEBUG::2014-09-25 18:17:32,522::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72079::ERROR::2014-09-25 18:17:32,522::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72079::DEBUG::2014-09-25 18:17:32,522::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:32,662::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:32,663::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72080::DEBUG::2014-09-25 18:17:32,665::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:32,995::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:33,046::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0297112 s, 34.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:17:35,068::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:35,103::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0189007 s, 54.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:35,679::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:35,680::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72081::DEBUG::2014-09-25 18:17:35,683::task::595::Storage.TaskManager.Task::(_updateState) Task=`f4204bdc-60a3-496e-a944-75cae2ef52a2`::moving from state init -> state preparing >Thread-72081::INFO::2014-09-25 18:17:35,683::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72081::INFO::2014-09-25 18:17:35,684::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00318308', 'lastCheck': '9.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '67.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00288537', 'lastCheck': '6.2', 'valid': True}} >Thread-72081::DEBUG::2014-09-25 18:17:35,684::task::1191::Storage.TaskManager.Task::(prepare) Task=`f4204bdc-60a3-496e-a944-75cae2ef52a2`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00318308', 'lastCheck': '9.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '67.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00288537', 'lastCheck': '6.2', 'valid': True}} >Thread-72081::DEBUG::2014-09-25 18:17:35,684::task::595::Storage.TaskManager.Task::(_updateState) Task=`f4204bdc-60a3-496e-a944-75cae2ef52a2`::moving from state preparing -> state finished >Thread-72081::DEBUG::2014-09-25 18:17:35,685::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72081::DEBUG::2014-09-25 18:17:35,685::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72081::DEBUG::2014-09-25 18:17:35,685::task::993::Storage.TaskManager.Task::(_decref) Task=`f4204bdc-60a3-496e-a944-75cae2ef52a2`::ref 0 aborting False >Thread-72081::DEBUG::2014-09-25 18:17:35,687::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:35,711::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:35,712::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72082::DEBUG::2014-09-25 18:17:35,714::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:17:36,293::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:36,293::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:36,294::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:36,294::__init__::232::IOProcess::(_processLogs) (111003) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:36,294::__init__::232::IOProcess::(_processLogs) (111003) Queuing response >Thread-79::DEBUG::2014-09-25 18:17:36,295::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:17:36,304::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000514949 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:36,305::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:36,306::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:36,306::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:36,306::__init__::232::IOProcess::(_processLogs) (111004) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:36,307::__init__::232::IOProcess::(_processLogs) (111004) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:36,308::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:36,309::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:36,309::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:36,309::__init__::232::IOProcess::(_processLogs) (111005) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:36,309::__init__::232::IOProcess::(_processLogs) (111005) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:36,310::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:36,311::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:36,311::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:36,312::__init__::232::IOProcess::(_processLogs) (111006) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:36,312::__init__::232::IOProcess::(_processLogs) (111006) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:36,312::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:36,313::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:36,313::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:36,314::__init__::232::IOProcess::(_processLogs) (111007) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:36,314::__init__::232::IOProcess::(_processLogs) (111007) Queuing response >Dummy-148::DEBUG::2014-09-25 18:17:37,123::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:37,149::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015406 s, 66.5 MB/s\n'; <rc> = 0 >Thread-72052::WARNING::2014-09-25 18:17:38,163::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:17:38,164::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:17:38,165::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:38,763::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:38,764::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72083::DEBUG::2014-09-25 18:17:38,766::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:39,169::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:39,202::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0196739 s, 52.0 MB/s\n'; <rc> = 0 >Thread-81::DEBUG::2014-09-25 18:17:39,484::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain 3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 18:17:39,488::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:39,489::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:39,489::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:39,489::__init__::232::IOProcess::(_processLogs) (111008) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:17:39,490::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:39,490::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:39,490::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:39,491::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:39,491::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:39,491::__init__::232::IOProcess::(_processLogs) (111009) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:17:39,491::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:39,492::__init__::232::IOProcess::(_processLogs) (111010) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:17:39,511::__init__::232::IOProcess::(_processLogs) (111008) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:39,513::__init__::232::IOProcess::(_processLogs) (111010) Queuing response >Thread-81::DEBUG::2014-09-25 18:17:39,517::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f >Thread-29::DEBUG::2014-09-25 18:17:39,518::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:39,518::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:39,519::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:39,519::__init__::232::IOProcess::(_processLogs) (111011) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 18:17:39,564::__init__::232::IOProcess::(_processLogs) (111011) Queuing response >Thread-81::DEBUG::2014-09-25 18:17:39,564::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:17:39,566::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:39,566::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:39,566::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:39,566::__init__::232::IOProcess::(_processLogs) (111012) Got request for method 'readfile' >Thread-81::DEBUG::2014-09-25 18:17:39,571::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Backup', 'DESCRIPTION=export', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=0', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35export', 'ROLE=Regular', 'SDUUID=3573f707-d0c4-4dc9-b09f-dca9a3506a4f', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=0604dd344a393112245d330869ebbc5bbee33ea3'] >Thread-29::DEBUG::2014-09-25 18:17:39,571::__init__::232::IOProcess::(_processLogs) (111012) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:39,573::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:39,574::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:39,574::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:39,574::__init__::232::IOProcess::(_processLogs) (111013) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:39,574::__init__::232::IOProcess::(_processLogs) (111013) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:39,575::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:39,575::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:39,576::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:39,576::__init__::232::IOProcess::(_processLogs) (111014) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:17:39,579::__init__::232::IOProcess::(_processLogs) (111014) Queuing response >Thread-81::DEBUG::2014-09-25 18:17:39,580::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-81::INFO::2014-09-25 18:17:39,581::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_imageNS already registered >Thread-81::INFO::2014-09-25 18:17:39,581::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace 3573f707-d0c4-4dc9-b09f-dca9a3506a4f_volumeNS already registered >Thread-29::DEBUG::2014-09-25 18:17:39,582::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:39,582::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:39,584::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:39,585::__init__::232::IOProcess::(_processLogs) (111015) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:39,588::__init__::232::IOProcess::(_processLogs) (111015) Queuing response >Thread-81::DEBUG::2014-09-25 18:17:39,590::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:17:39,603::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00230845 s, 161 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:39,606::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:39,606::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:39,607::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:39,607::__init__::232::IOProcess::(_processLogs) (111016) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:39,608::__init__::232::IOProcess::(_processLogs) (111016) Queuing response >Dummy-148::DEBUG::2014-09-25 18:17:41,222::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:41,254::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200458 s, 51.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:41,805::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:41,807::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72087::DEBUG::2014-09-25 18:17:41,809::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:42,547::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:42,548::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72088::DEBUG::2014-09-25 18:17:42,549::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72088::DEBUG::2014-09-25 18:17:42,550::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::moving from state init -> state preparing >Thread-72088::INFO::2014-09-25 18:17:42,551::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72088::ERROR::2014-09-25 18:17:42,551::task::866::Storage.TaskManager.Task::(_setError) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72088::DEBUG::2014-09-25 18:17:42,552::task::885::Storage.TaskManager.Task::(_run) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::Task._run: fc899e81-a569-4cb8-9e25-39009b3d8873 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72088::DEBUG::2014-09-25 18:17:42,552::task::1217::Storage.TaskManager.Task::(stop) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::stopping in state preparing (force False) >Thread-72088::DEBUG::2014-09-25 18:17:42,552::task::993::Storage.TaskManager.Task::(_decref) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::ref 1 aborting True >Thread-72088::INFO::2014-09-25 18:17:42,552::task::1171::Storage.TaskManager.Task::(prepare) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::aborting: Task is aborted: u'' - code 100 >Thread-72088::DEBUG::2014-09-25 18:17:42,553::task::1176::Storage.TaskManager.Task::(prepare) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::Prepare: aborted: >Thread-72088::DEBUG::2014-09-25 18:17:42,553::task::993::Storage.TaskManager.Task::(_decref) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::ref 0 aborting True >Thread-72088::DEBUG::2014-09-25 18:17:42,553::task::928::Storage.TaskManager.Task::(_doAbort) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::Task._doAbort: force False >Thread-72088::DEBUG::2014-09-25 18:17:42,553::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72088::DEBUG::2014-09-25 18:17:42,553::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::moving from state preparing -> state aborting >Thread-72088::DEBUG::2014-09-25 18:17:42,554::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::_aborting: recover policy none >Thread-72088::DEBUG::2014-09-25 18:17:42,554::task::595::Storage.TaskManager.Task::(_updateState) Task=`fc899e81-a569-4cb8-9e25-39009b3d8873`::moving from state aborting -> state failed >Thread-72088::DEBUG::2014-09-25 18:17:42,554::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72088::DEBUG::2014-09-25 18:17:42,554::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72088::ERROR::2014-09-25 18:17:42,554::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72088::DEBUG::2014-09-25 18:17:42,555::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:43,274::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:43,299::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147088 s, 69.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:44,822::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:44,823::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72089::DEBUG::2014-09-25 18:17:44,826::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:45,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:45,346::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0161376 s, 63.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:46,316::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:46,318::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-79::DEBUG::2014-09-25 18:17:46,320::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:17:46,321::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:46,331::__init__::232::IOProcess::(_processLogs) (111017) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 18:17:46,332::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000364138 s, 1.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:46,334::__init__::232::IOProcess::(_processLogs) (111017) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:46,336::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:46,337::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:46,338::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:46,339::__init__::232::IOProcess::(_processLogs) (111018) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:46,341::__init__::232::IOProcess::(_processLogs) (111018) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:46,344::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:46,346::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:46,347::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:46,348::__init__::232::IOProcess::(_processLogs) (111019) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:46,349::__init__::232::IOProcess::(_processLogs) (111019) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:46,350::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:46,351::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:46,352::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:46,353::__init__::232::IOProcess::(_processLogs) (111020) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:46,355::__init__::232::IOProcess::(_processLogs) (111020) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:46,358::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:46,359::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:46,360::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:46,361::__init__::232::IOProcess::(_processLogs) (111021) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:46,362::__init__::232::IOProcess::(_processLogs) (111021) Queuing response >Dummy-148::DEBUG::2014-09-25 18:17:47,366::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:47,396::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201377 s, 50.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:47,840::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:47,841::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72090::DEBUG::2014-09-25 18:17:47,843::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:17:48,169::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:48,170::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:48,170::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:48,171::__init__::232::IOProcess::(_processLogs) (111022) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:17:48,171::__init__::232::IOProcess::(_processLogs) (111022) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:48,172::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:48,173::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:48,173::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:48,173::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:48,173::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:48,174::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:48,174::__init__::232::IOProcess::(_processLogs) (111023) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:17:48,174::__init__::232::IOProcess::(_processLogs) (111024) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:17:48,175::__init__::232::IOProcess::(_processLogs) (111024) Queuing response >Dummy-148::DEBUG::2014-09-25 18:17:49,414::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:49,444::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0162315 s, 63.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:49,609::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:49,610::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:49,610::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:49,610::__init__::232::IOProcess::(_processLogs) (111025) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:49,615::__init__::232::IOProcess::(_processLogs) (111025) Queuing response >Thread-81::DEBUG::2014-09-25 18:17:49,616::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:17:49,631::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00341757 s, 109 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:49,633::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:49,633::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:49,633::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:49,633::__init__::232::IOProcess::(_processLogs) (111026) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:49,635::__init__::232::IOProcess::(_processLogs) (111026) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:50,857::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:50,859::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72094::DEBUG::2014-09-25 18:17:50,862::task::595::Storage.TaskManager.Task::(_updateState) Task=`1f87900b-cc11-44ff-a898-8b3c045a3a9a`::moving from state init -> state preparing >Thread-72094::INFO::2014-09-25 18:17:50,862::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72094::INFO::2014-09-25 18:17:50,863::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000364138', 'lastCheck': '4.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '12.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00341757', 'lastCheck': '1.2', 'valid': True}} >Thread-72094::DEBUG::2014-09-25 18:17:50,863::task::1191::Storage.TaskManager.Task::(prepare) Task=`1f87900b-cc11-44ff-a898-8b3c045a3a9a`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000364138', 'lastCheck': '4.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '12.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00341757', 'lastCheck': '1.2', 'valid': True}} >Thread-72094::DEBUG::2014-09-25 18:17:50,863::task::595::Storage.TaskManager.Task::(_updateState) Task=`1f87900b-cc11-44ff-a898-8b3c045a3a9a`::moving from state preparing -> state finished >Thread-72094::DEBUG::2014-09-25 18:17:50,863::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72094::DEBUG::2014-09-25 18:17:50,864::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72094::DEBUG::2014-09-25 18:17:50,864::task::993::Storage.TaskManager.Task::(_decref) Task=`1f87900b-cc11-44ff-a898-8b3c045a3a9a`::ref 0 aborting False >Thread-72094::DEBUG::2014-09-25 18:17:50,866::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:50,886::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:50,886::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72095::DEBUG::2014-09-25 18:17:50,888::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:51,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:51,495::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0199188 s, 51.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:52,586::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:52,588::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72096::DEBUG::2014-09-25 18:17:52,588::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72096::DEBUG::2014-09-25 18:17:52,589::task::595::Storage.TaskManager.Task::(_updateState) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::moving from state init -> state preparing >Thread-72096::INFO::2014-09-25 18:17:52,590::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72096::ERROR::2014-09-25 18:17:52,590::task::866::Storage.TaskManager.Task::(_setError) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72096::DEBUG::2014-09-25 18:17:52,591::task::885::Storage.TaskManager.Task::(_run) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::Task._run: 9256b95a-c378-4384-a739-1e8a5ba45c50 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72096::DEBUG::2014-09-25 18:17:52,591::task::1217::Storage.TaskManager.Task::(stop) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::stopping in state preparing (force False) >Thread-72096::DEBUG::2014-09-25 18:17:52,591::task::993::Storage.TaskManager.Task::(_decref) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::ref 1 aborting True >Thread-72096::INFO::2014-09-25 18:17:52,591::task::1171::Storage.TaskManager.Task::(prepare) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::aborting: Task is aborted: u'' - code 100 >Thread-72096::DEBUG::2014-09-25 18:17:52,591::task::1176::Storage.TaskManager.Task::(prepare) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::Prepare: aborted: >Thread-72096::DEBUG::2014-09-25 18:17:52,592::task::993::Storage.TaskManager.Task::(_decref) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::ref 0 aborting True >Thread-72096::DEBUG::2014-09-25 18:17:52,592::task::928::Storage.TaskManager.Task::(_doAbort) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::Task._doAbort: force False >Thread-72096::DEBUG::2014-09-25 18:17:52,592::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72096::DEBUG::2014-09-25 18:17:52,592::task::595::Storage.TaskManager.Task::(_updateState) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::moving from state preparing -> state aborting >Thread-72096::DEBUG::2014-09-25 18:17:52,593::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::_aborting: recover policy none >Thread-72096::DEBUG::2014-09-25 18:17:52,593::task::595::Storage.TaskManager.Task::(_updateState) Task=`9256b95a-c378-4384-a739-1e8a5ba45c50`::moving from state aborting -> state failed >Thread-72096::DEBUG::2014-09-25 18:17:52,593::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72096::DEBUG::2014-09-25 18:17:52,593::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72096::ERROR::2014-09-25 18:17:52,593::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72096::DEBUG::2014-09-25 18:17:52,594::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:53,515::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:53,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213117 s, 48.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:53,946::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:53,947::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72097::DEBUG::2014-09-25 18:17:53,949::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:55,571::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:55,610::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207578 s, 49.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:56,360::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:56,360::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:56,360::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:56,360::__init__::232::IOProcess::(_processLogs) (111027) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:56,361::__init__::232::IOProcess::(_processLogs) (111027) Queuing response >Thread-79::DEBUG::2014-09-25 18:17:56,362::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:17:56,370::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000437366 s, 1.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:56,372::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:56,372::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:56,372::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:56,373::__init__::232::IOProcess::(_processLogs) (111028) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:56,373::__init__::232::IOProcess::(_processLogs) (111028) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:56,374::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:56,375::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:56,375::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:56,375::__init__::232::IOProcess::(_processLogs) (111029) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:56,376::__init__::232::IOProcess::(_processLogs) (111029) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:56,377::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:56,377::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:56,377::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:56,377::__init__::232::IOProcess::(_processLogs) (111030) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:56,378::__init__::232::IOProcess::(_processLogs) (111030) Queuing response >Thread-29::DEBUG::2014-09-25 18:17:56,379::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:56,379::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:56,379::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:56,379::__init__::232::IOProcess::(_processLogs) (111031) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:17:56,380::__init__::232::IOProcess::(_processLogs) (111031) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:56,963::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:56,964::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72098::DEBUG::2014-09-25 18:17:56,966::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:17:57,630::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:57,658::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168396 s, 60.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:59,639::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:59,639::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:59,639::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:59,639::__init__::232::IOProcess::(_processLogs) (111032) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:59,642::__init__::232::IOProcess::(_processLogs) (111032) Queuing response >Thread-81::DEBUG::2014-09-25 18:17:59,642::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:17:59,658::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00458552 s, 80.9 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:17:59,659::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:17:59,659::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:17:59,659::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:17:59,660::__init__::232::IOProcess::(_processLogs) (111033) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:17:59,662::__init__::232::IOProcess::(_processLogs) (111033) Queuing response >Dummy-148::DEBUG::2014-09-25 18:17:59,691::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:17:59,720::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0186568 s, 54.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:17:59,981::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:17:59,983::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72099::DEBUG::2014-09-25 18:17:59,984::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:01,740::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:01,768::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167869 s, 61.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:02,628::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72100::DEBUG::2014-09-25 18:18:02,629::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:18:02,629::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72100::DEBUG::2014-09-25 18:18:02,631::task::595::Storage.TaskManager.Task::(_updateState) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::moving from state init -> state preparing >Thread-72100::INFO::2014-09-25 18:18:02,632::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72100::ERROR::2014-09-25 18:18:02,632::task::866::Storage.TaskManager.Task::(_setError) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72100::DEBUG::2014-09-25 18:18:02,633::task::885::Storage.TaskManager.Task::(_run) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::Task._run: 71d5f633-0a11-4008-a01f-d81b98f347e0 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72100::DEBUG::2014-09-25 18:18:02,633::task::1217::Storage.TaskManager.Task::(stop) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::stopping in state preparing (force False) >Thread-72100::DEBUG::2014-09-25 18:18:02,633::task::993::Storage.TaskManager.Task::(_decref) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::ref 1 aborting True >Thread-72100::INFO::2014-09-25 18:18:02,633::task::1171::Storage.TaskManager.Task::(prepare) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::aborting: Task is aborted: u'' - code 100 >Thread-72100::DEBUG::2014-09-25 18:18:02,634::task::1176::Storage.TaskManager.Task::(prepare) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::Prepare: aborted: >Thread-72100::DEBUG::2014-09-25 18:18:02,634::task::993::Storage.TaskManager.Task::(_decref) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::ref 0 aborting True >Thread-72100::DEBUG::2014-09-25 18:18:02,634::task::928::Storage.TaskManager.Task::(_doAbort) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::Task._doAbort: force False >Thread-72100::DEBUG::2014-09-25 18:18:02,634::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72100::DEBUG::2014-09-25 18:18:02,635::task::595::Storage.TaskManager.Task::(_updateState) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::moving from state preparing -> state aborting >Thread-72100::DEBUG::2014-09-25 18:18:02,635::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::_aborting: recover policy none >Thread-72100::DEBUG::2014-09-25 18:18:02,635::task::595::Storage.TaskManager.Task::(_updateState) Task=`71d5f633-0a11-4008-a01f-d81b98f347e0`::moving from state aborting -> state failed >Thread-72100::DEBUG::2014-09-25 18:18:02,635::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72100::DEBUG::2014-09-25 18:18:02,635::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72100::ERROR::2014-09-25 18:18:02,636::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72100::DEBUG::2014-09-25 18:18:02,636::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:02,997::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:02,998::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72101::DEBUG::2014-09-25 18:18:03,000::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:03,788::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:03,811::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0138274 s, 74.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:18:05,831::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:05,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158592 s, 64.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:06,014::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:06,016::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72102::DEBUG::2014-09-25 18:18:06,019::task::595::Storage.TaskManager.Task::(_updateState) Task=`aec8a92f-f3f8-4bbb-9ded-44f119a85cfd`::moving from state init -> state preparing >Thread-72102::INFO::2014-09-25 18:18:06,019::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72102::INFO::2014-09-25 18:18:06,020::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000437366', 'lastCheck': '9.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '27.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00458552', 'lastCheck': '6.4', 'valid': True}} >Thread-72102::DEBUG::2014-09-25 18:18:06,020::task::1191::Storage.TaskManager.Task::(prepare) Task=`aec8a92f-f3f8-4bbb-9ded-44f119a85cfd`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000437366', 'lastCheck': '9.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '27.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00458552', 'lastCheck': '6.4', 'valid': True}} >Thread-72102::DEBUG::2014-09-25 18:18:06,020::task::595::Storage.TaskManager.Task::(_updateState) Task=`aec8a92f-f3f8-4bbb-9ded-44f119a85cfd`::moving from state preparing -> state finished >Thread-72102::DEBUG::2014-09-25 18:18:06,020::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72102::DEBUG::2014-09-25 18:18:06,020::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72102::DEBUG::2014-09-25 18:18:06,021::task::993::Storage.TaskManager.Task::(_decref) Task=`aec8a92f-f3f8-4bbb-9ded-44f119a85cfd`::ref 0 aborting False >Thread-72102::DEBUG::2014-09-25 18:18:06,023::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:06,043::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:06,044::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72103::DEBUG::2014-09-25 18:18:06,046::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:18:06,382::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:06,382::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:06,382::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:06,383::__init__::232::IOProcess::(_processLogs) (111034) Got request for method 'statvfs' >Thread-79::DEBUG::2014-09-25 18:18:06,384::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-29::DEBUG::2014-09-25 18:18:06,384::__init__::232::IOProcess::(_processLogs) (111034) Queuing response >Thread-79::DEBUG::2014-09-25 18:18:06,393::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000342394 s, 1.6 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:06,394::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:06,395::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:06,395::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:06,396::__init__::232::IOProcess::(_processLogs) (111035) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:06,396::__init__::232::IOProcess::(_processLogs) (111035) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:06,397::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:06,397::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:06,398::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:06,398::__init__::232::IOProcess::(_processLogs) (111036) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:06,398::__init__::232::IOProcess::(_processLogs) (111036) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:06,399::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:06,399::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:06,400::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:06,400::__init__::232::IOProcess::(_processLogs) (111037) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:06,400::__init__::232::IOProcess::(_processLogs) (111037) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:06,401::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:06,401::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:06,402::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:06,403::__init__::232::IOProcess::(_processLogs) (111038) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:06,403::__init__::232::IOProcess::(_processLogs) (111038) Queuing response >Dummy-148::DEBUG::2014-09-25 18:18:07,878::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:07,901::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135031 s, 75.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:09,100::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:09,101::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72104::DEBUG::2014-09-25 18:18:09,103::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:18:09,664::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:09,664::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:09,664::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:09,665::__init__::232::IOProcess::(_processLogs) (111039) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:09,674::__init__::232::IOProcess::(_processLogs) (111039) Queuing response >Thread-81::DEBUG::2014-09-25 18:18:09,676::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:18:09,702::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00366194 s, 101 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:09,703::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:09,703::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:09,704::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:09,704::__init__::232::IOProcess::(_processLogs) (111040) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:09,708::__init__::232::IOProcess::(_processLogs) (111040) Queuing response >Dummy-148::DEBUG::2014-09-25 18:18:09,920::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:09,943::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133282 s, 76.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:18:11,962::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:11,985::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134826 s, 75.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:12,123::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:12,125::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72105::DEBUG::2014-09-25 18:18:12,127::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:12,659::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72106::DEBUG::2014-09-25 18:18:12,660::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:18:12,660::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72106::DEBUG::2014-09-25 18:18:12,663::task::595::Storage.TaskManager.Task::(_updateState) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::moving from state init -> state preparing >Thread-72106::INFO::2014-09-25 18:18:12,663::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72106::ERROR::2014-09-25 18:18:12,664::task::866::Storage.TaskManager.Task::(_setError) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72106::DEBUG::2014-09-25 18:18:12,665::task::885::Storage.TaskManager.Task::(_run) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::Task._run: 457d87f0-1625-4a8f-9e0c-6d86371e3baf (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72106::DEBUG::2014-09-25 18:18:12,665::task::1217::Storage.TaskManager.Task::(stop) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::stopping in state preparing (force False) >Thread-72106::DEBUG::2014-09-25 18:18:12,666::task::993::Storage.TaskManager.Task::(_decref) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::ref 1 aborting True >Thread-72106::INFO::2014-09-25 18:18:12,666::task::1171::Storage.TaskManager.Task::(prepare) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::aborting: Task is aborted: u'' - code 100 >Thread-72106::DEBUG::2014-09-25 18:18:12,666::task::1176::Storage.TaskManager.Task::(prepare) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::Prepare: aborted: >Thread-72106::DEBUG::2014-09-25 18:18:12,666::task::993::Storage.TaskManager.Task::(_decref) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::ref 0 aborting True >Thread-72106::DEBUG::2014-09-25 18:18:12,666::task::928::Storage.TaskManager.Task::(_doAbort) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::Task._doAbort: force False >Thread-72106::DEBUG::2014-09-25 18:18:12,667::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72106::DEBUG::2014-09-25 18:18:12,667::task::595::Storage.TaskManager.Task::(_updateState) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::moving from state preparing -> state aborting >Thread-72106::DEBUG::2014-09-25 18:18:12,667::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::_aborting: recover policy none >Thread-72106::DEBUG::2014-09-25 18:18:12,667::task::595::Storage.TaskManager.Task::(_updateState) Task=`457d87f0-1625-4a8f-9e0c-6d86371e3baf`::moving from state aborting -> state failed >Thread-72106::DEBUG::2014-09-25 18:18:12,667::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72106::DEBUG::2014-09-25 18:18:12,668::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72106::ERROR::2014-09-25 18:18:12,668::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72106::DEBUG::2014-09-25 18:18:12,668::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:14,003::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:14,030::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174755 s, 58.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:15,140::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:15,142::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72107::DEBUG::2014-09-25 18:18:15,144::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:16,051::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:16,092::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0284645 s, 36.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:16,404::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:16,405::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:16,406::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:16,407::__init__::232::IOProcess::(_processLogs) (111041) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:16,409::__init__::232::IOProcess::(_processLogs) (111041) Queuing response >Thread-79::DEBUG::2014-09-25 18:18:16,410::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:18:16,421::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000716706 s, 784 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:16,424::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:16,425::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:16,426::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:16,427::__init__::232::IOProcess::(_processLogs) (111042) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:16,429::__init__::232::IOProcess::(_processLogs) (111042) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:16,431::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:16,432::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:16,433::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:16,434::__init__::232::IOProcess::(_processLogs) (111043) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:16,435::__init__::232::IOProcess::(_processLogs) (111043) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:16,438::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:16,439::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:16,440::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:16,441::__init__::232::IOProcess::(_processLogs) (111044) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:16,442::__init__::232::IOProcess::(_processLogs) (111044) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:16,444::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:16,445::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:16,447::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:16,448::__init__::232::IOProcess::(_processLogs) (111045) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:16,449::__init__::232::IOProcess::(_processLogs) (111045) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:18,158::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:18,159::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72108::DEBUG::2014-09-25 18:18:18,161::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:18,290::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:18,315::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0157385 s, 65.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:19,710::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:19,711::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:19,711::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:19,711::__init__::232::IOProcess::(_processLogs) (111046) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:19,715::__init__::232::IOProcess::(_processLogs) (111046) Queuing response >Thread-81::DEBUG::2014-09-25 18:18:19,717::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:18:19,731::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00184281 s, 201 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:19,732::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:19,733::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:19,733::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:19,733::__init__::232::IOProcess::(_processLogs) (111047) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:19,739::__init__::232::IOProcess::(_processLogs) (111047) Queuing response >Dummy-148::DEBUG::2014-09-25 18:18:20,335::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:20,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0155803 s, 65.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:21,177::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:21,179::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72109::DEBUG::2014-09-25 18:18:21,181::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3e85463-c5b8-432f-b0d0-8d69f8004da2`::moving from state init -> state preparing >Thread-72109::INFO::2014-09-25 18:18:21,182::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72109::INFO::2014-09-25 18:18:21,182::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000716706', 'lastCheck': '4.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '43.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00184281', 'lastCheck': '1.4', 'valid': True}} >Thread-72109::DEBUG::2014-09-25 18:18:21,182::task::1191::Storage.TaskManager.Task::(prepare) Task=`a3e85463-c5b8-432f-b0d0-8d69f8004da2`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000716706', 'lastCheck': '4.7', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '43.0', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00184281', 'lastCheck': '1.4', 'valid': True}} >Thread-72109::DEBUG::2014-09-25 18:18:21,183::task::595::Storage.TaskManager.Task::(_updateState) Task=`a3e85463-c5b8-432f-b0d0-8d69f8004da2`::moving from state preparing -> state finished >Thread-72109::DEBUG::2014-09-25 18:18:21,183::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72109::DEBUG::2014-09-25 18:18:21,183::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72109::DEBUG::2014-09-25 18:18:21,183::task::993::Storage.TaskManager.Task::(_decref) Task=`a3e85463-c5b8-432f-b0d0-8d69f8004da2`::ref 0 aborting False >Thread-72109::DEBUG::2014-09-25 18:18:21,186::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:21,203::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:21,204::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72110::DEBUG::2014-09-25 18:18:21,206::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:22,381::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:22,419::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0212078 s, 48.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:22,701::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72111::DEBUG::2014-09-25 18:18:22,703::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:18:22,703::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72111::DEBUG::2014-09-25 18:18:22,705::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::moving from state init -> state preparing >Thread-72111::INFO::2014-09-25 18:18:22,706::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72111::ERROR::2014-09-25 18:18:22,706::task::866::Storage.TaskManager.Task::(_setError) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72111::DEBUG::2014-09-25 18:18:22,706::task::885::Storage.TaskManager.Task::(_run) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::Task._run: 3b836eff-6126-4abb-b687-7ab820c42338 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72111::DEBUG::2014-09-25 18:18:22,707::task::1217::Storage.TaskManager.Task::(stop) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::stopping in state preparing (force False) >Thread-72111::DEBUG::2014-09-25 18:18:22,707::task::993::Storage.TaskManager.Task::(_decref) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::ref 1 aborting True >Thread-72111::INFO::2014-09-25 18:18:22,707::task::1171::Storage.TaskManager.Task::(prepare) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::aborting: Task is aborted: u'' - code 100 >Thread-72111::DEBUG::2014-09-25 18:18:22,707::task::1176::Storage.TaskManager.Task::(prepare) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::Prepare: aborted: >Thread-72111::DEBUG::2014-09-25 18:18:22,708::task::993::Storage.TaskManager.Task::(_decref) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::ref 0 aborting True >Thread-72111::DEBUG::2014-09-25 18:18:22,708::task::928::Storage.TaskManager.Task::(_doAbort) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::Task._doAbort: force False >Thread-72111::DEBUG::2014-09-25 18:18:22,708::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72111::DEBUG::2014-09-25 18:18:22,708::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::moving from state preparing -> state aborting >Thread-72111::DEBUG::2014-09-25 18:18:22,708::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::_aborting: recover policy none >Thread-72111::DEBUG::2014-09-25 18:18:22,709::task::595::Storage.TaskManager.Task::(_updateState) Task=`3b836eff-6126-4abb-b687-7ab820c42338`::moving from state aborting -> state failed >Thread-72111::DEBUG::2014-09-25 18:18:22,709::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72111::DEBUG::2014-09-25 18:18:22,709::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72111::ERROR::2014-09-25 18:18:22,709::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72111::DEBUG::2014-09-25 18:18:22,710::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:24,255::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:24,256::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72112::DEBUG::2014-09-25 18:18:24,258::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:24,439::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:24,461::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134164 s, 76.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:26,449::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:26,449::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:26,449::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:26,450::__init__::232::IOProcess::(_processLogs) (111048) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:26,450::__init__::232::IOProcess::(_processLogs) (111048) Queuing response >Thread-79::DEBUG::2014-09-25 18:18:26,451::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:18:26,460::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000327156 s, 1.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:26,461::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:26,462::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:26,462::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:26,462::__init__::232::IOProcess::(_processLogs) (111049) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:26,462::__init__::232::IOProcess::(_processLogs) (111049) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:26,463::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:26,464::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:26,465::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:26,465::__init__::232::IOProcess::(_processLogs) (111050) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:26,465::__init__::232::IOProcess::(_processLogs) (111050) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:26,465::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:26,466::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:26,467::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:26,467::__init__::232::IOProcess::(_processLogs) (111051) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:26,467::__init__::232::IOProcess::(_processLogs) (111051) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:26,468::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:26,468::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:26,469::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:26,469::__init__::232::IOProcess::(_processLogs) (111052) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:26,470::__init__::232::IOProcess::(_processLogs) (111052) Queuing response >Dummy-148::DEBUG::2014-09-25 18:18:26,481::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:26,502::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013069 s, 78.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:27,275::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:27,277::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72113::DEBUG::2014-09-25 18:18:27,279::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:28,522::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:28,548::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015222 s, 67.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:29,740::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:29,741::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:29,741::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:29,741::__init__::232::IOProcess::(_processLogs) (111053) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:29,743::__init__::232::IOProcess::(_processLogs) (111053) Queuing response >Thread-81::DEBUG::2014-09-25 18:18:29,744::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:18:29,766::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00696121 s, 53.3 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:29,767::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:29,767::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:29,768::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:29,768::__init__::232::IOProcess::(_processLogs) (111054) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:29,774::__init__::232::IOProcess::(_processLogs) (111054) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:30,294::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:30,296::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72114::DEBUG::2014-09-25 18:18:30,297::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:30,568::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:30,597::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015915 s, 64.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:18:32,616::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:32,647::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159761 s, 64.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:32,734::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:32,736::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72115::DEBUG::2014-09-25 18:18:32,736::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72115::DEBUG::2014-09-25 18:18:32,738::task::595::Storage.TaskManager.Task::(_updateState) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::moving from state init -> state preparing >Thread-72115::INFO::2014-09-25 18:18:32,739::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72115::ERROR::2014-09-25 18:18:32,739::task::866::Storage.TaskManager.Task::(_setError) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72115::DEBUG::2014-09-25 18:18:32,740::task::885::Storage.TaskManager.Task::(_run) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::Task._run: c9e983b9-48f8-4a4c-848b-7bd3e686e854 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72115::DEBUG::2014-09-25 18:18:32,740::task::1217::Storage.TaskManager.Task::(stop) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::stopping in state preparing (force False) >Thread-72115::DEBUG::2014-09-25 18:18:32,740::task::993::Storage.TaskManager.Task::(_decref) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::ref 1 aborting True >Thread-72115::INFO::2014-09-25 18:18:32,740::task::1171::Storage.TaskManager.Task::(prepare) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::aborting: Task is aborted: u'' - code 100 >Thread-72115::DEBUG::2014-09-25 18:18:32,740::task::1176::Storage.TaskManager.Task::(prepare) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::Prepare: aborted: >Thread-72115::DEBUG::2014-09-25 18:18:32,741::task::993::Storage.TaskManager.Task::(_decref) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::ref 0 aborting True >Thread-72115::DEBUG::2014-09-25 18:18:32,741::task::928::Storage.TaskManager.Task::(_doAbort) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::Task._doAbort: force False >Thread-72115::DEBUG::2014-09-25 18:18:32,741::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72115::DEBUG::2014-09-25 18:18:32,741::task::595::Storage.TaskManager.Task::(_updateState) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::moving from state preparing -> state aborting >Thread-72115::DEBUG::2014-09-25 18:18:32,742::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::_aborting: recover policy none >Thread-72115::DEBUG::2014-09-25 18:18:32,742::task::595::Storage.TaskManager.Task::(_updateState) Task=`c9e983b9-48f8-4a4c-848b-7bd3e686e854`::moving from state aborting -> state failed >Thread-72115::DEBUG::2014-09-25 18:18:32,742::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72115::DEBUG::2014-09-25 18:18:32,742::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72115::ERROR::2014-09-25 18:18:32,742::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72115::DEBUG::2014-09-25 18:18:32,743::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:33,312::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:33,313::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72116::DEBUG::2014-09-25 18:18:33,315::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:34,666::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:34,699::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.02007 s, 51.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:36,329::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:36,331::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72117::DEBUG::2014-09-25 18:18:36,333::task::595::Storage.TaskManager.Task::(_updateState) Task=`537609ad-825e-46cf-8f90-587b6eba4623`::moving from state init -> state preparing >Thread-72117::INFO::2014-09-25 18:18:36,334::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72117::INFO::2014-09-25 18:18:36,334::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000327156', 'lastCheck': '9.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '58.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00696121', 'lastCheck': '6.6', 'valid': True}} >Thread-72117::DEBUG::2014-09-25 18:18:36,334::task::1191::Storage.TaskManager.Task::(prepare) Task=`537609ad-825e-46cf-8f90-587b6eba4623`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000327156', 'lastCheck': '9.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '58.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00696121', 'lastCheck': '6.6', 'valid': True}} >Thread-72117::DEBUG::2014-09-25 18:18:36,335::task::595::Storage.TaskManager.Task::(_updateState) Task=`537609ad-825e-46cf-8f90-587b6eba4623`::moving from state preparing -> state finished >Thread-72117::DEBUG::2014-09-25 18:18:36,335::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72117::DEBUG::2014-09-25 18:18:36,335::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72117::DEBUG::2014-09-25 18:18:36,335::task::993::Storage.TaskManager.Task::(_decref) Task=`537609ad-825e-46cf-8f90-587b6eba4623`::ref 0 aborting False >Thread-72117::DEBUG::2014-09-25 18:18:36,337::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:36,353::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:36,354::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72118::DEBUG::2014-09-25 18:18:36,356::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:18:36,470::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:36,470::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:36,470::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:36,471::__init__::232::IOProcess::(_processLogs) (111055) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:36,473::__init__::232::IOProcess::(_processLogs) (111055) Queuing response >Thread-79::DEBUG::2014-09-25 18:18:36,474::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:18:36,496::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00897382 s, 62.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:36,497::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:36,498::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:36,498::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:36,498::__init__::232::IOProcess::(_processLogs) (111056) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:36,503::__init__::232::IOProcess::(_processLogs) (111056) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:36,505::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:36,505::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:36,505::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:36,505::__init__::232::IOProcess::(_processLogs) (111057) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:36,506::__init__::232::IOProcess::(_processLogs) (111057) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:36,506::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:36,507::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:36,507::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:36,507::__init__::232::IOProcess::(_processLogs) (111058) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:36,507::__init__::232::IOProcess::(_processLogs) (111058) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:36,508::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:36,508::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:36,509::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:36,509::__init__::232::IOProcess::(_processLogs) (111059) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:36,509::__init__::232::IOProcess::(_processLogs) (111059) Queuing response >Dummy-148::DEBUG::2014-09-25 18:18:36,719::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:36,760::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0220097 s, 46.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:18:38,780::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:38,820::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0246433 s, 41.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:39,404::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:39,405::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72119::DEBUG::2014-09-25 18:18:39,407::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-72085::WARNING::2014-09-25 18:18:39,487::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-29::DEBUG::2014-09-25 18:18:39,777::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:39,777::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:39,777::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:39,778::__init__::232::IOProcess::(_processLogs) (111060) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:39,780::__init__::232::IOProcess::(_processLogs) (111060) Queuing response >Thread-81::DEBUG::2014-09-25 18:18:39,782::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:18:39,800::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00849447 s, 43.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:39,801::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:39,801::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:39,802::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:39,802::__init__::232::IOProcess::(_processLogs) (111061) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:39,804::__init__::232::IOProcess::(_processLogs) (111061) Queuing response >Dummy-148::DEBUG::2014-09-25 18:18:40,840::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:40,869::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0167271 s, 61.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:42,420::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:42,422::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72120::DEBUG::2014-09-25 18:18:42,424::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:42,768::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72121::DEBUG::2014-09-25 18:18:42,769::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:18:42,769::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72121::DEBUG::2014-09-25 18:18:42,771::task::595::Storage.TaskManager.Task::(_updateState) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::moving from state init -> state preparing >Thread-72121::INFO::2014-09-25 18:18:42,771::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72121::ERROR::2014-09-25 18:18:42,772::task::866::Storage.TaskManager.Task::(_setError) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72121::DEBUG::2014-09-25 18:18:42,772::task::885::Storage.TaskManager.Task::(_run) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::Task._run: b490f1e8-4139-4488-bd6a-f65cbeb6f2ec (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72121::DEBUG::2014-09-25 18:18:42,773::task::1217::Storage.TaskManager.Task::(stop) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::stopping in state preparing (force False) >Thread-72121::DEBUG::2014-09-25 18:18:42,773::task::993::Storage.TaskManager.Task::(_decref) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::ref 1 aborting True >Thread-72121::INFO::2014-09-25 18:18:42,773::task::1171::Storage.TaskManager.Task::(prepare) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::aborting: Task is aborted: u'' - code 100 >Thread-72121::DEBUG::2014-09-25 18:18:42,773::task::1176::Storage.TaskManager.Task::(prepare) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::Prepare: aborted: >Thread-72121::DEBUG::2014-09-25 18:18:42,773::task::993::Storage.TaskManager.Task::(_decref) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::ref 0 aborting True >Thread-72121::DEBUG::2014-09-25 18:18:42,774::task::928::Storage.TaskManager.Task::(_doAbort) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::Task._doAbort: force False >Thread-72121::DEBUG::2014-09-25 18:18:42,774::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72121::DEBUG::2014-09-25 18:18:42,774::task::595::Storage.TaskManager.Task::(_updateState) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::moving from state preparing -> state aborting >Thread-72121::DEBUG::2014-09-25 18:18:42,774::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::_aborting: recover policy none >Thread-72121::DEBUG::2014-09-25 18:18:42,775::task::595::Storage.TaskManager.Task::(_updateState) Task=`b490f1e8-4139-4488-bd6a-f65cbeb6f2ec`::moving from state aborting -> state failed >Thread-72121::DEBUG::2014-09-25 18:18:42,775::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72121::DEBUG::2014-09-25 18:18:42,775::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72121::ERROR::2014-09-25 18:18:42,775::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72121::DEBUG::2014-09-25 18:18:42,776::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:42,889::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:42,915::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0140075 s, 73.1 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:18:44,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:45,171::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.220189 s, 4.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:45,443::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:45,445::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72122::DEBUG::2014-09-25 18:18:45,447::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:18:46,512::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:46,513::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:46,514::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:46,516::__init__::232::IOProcess::(_processLogs) (111062) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:46,518::__init__::232::IOProcess::(_processLogs) (111062) Queuing response >Thread-79::DEBUG::2014-09-25 18:18:46,520::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:18:46,535::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00165142 s, 340 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:46,539::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:46,540::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:46,541::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:46,543::__init__::232::IOProcess::(_processLogs) (111063) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:46,544::__init__::232::IOProcess::(_processLogs) (111063) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:46,547::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:46,549::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:46,550::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:46,551::__init__::232::IOProcess::(_processLogs) (111064) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:46,552::__init__::232::IOProcess::(_processLogs) (111064) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:46,555::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:46,557::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:46,558::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:46,559::__init__::232::IOProcess::(_processLogs) (111065) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:46,560::__init__::232::IOProcess::(_processLogs) (111065) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:46,564::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:46,566::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:46,567::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:46,568::__init__::232::IOProcess::(_processLogs) (111066) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:46,569::__init__::232::IOProcess::(_processLogs) (111066) Queuing response >Dummy-148::DEBUG::2014-09-25 18:18:47,192::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:47,243::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0311183 s, 32.9 MB/s\n'; <rc> = 0 >Thread-72092::WARNING::2014-09-25 18:18:48,169::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:18:48,170::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:18:48,171::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:48,461::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:48,462::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72123::DEBUG::2014-09-25 18:18:48,464::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:49,263::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:49,296::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0222566 s, 46.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:49,806::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:49,807::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:49,807::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:49,807::__init__::232::IOProcess::(_processLogs) (111067) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:49,810::__init__::232::IOProcess::(_processLogs) (111067) Queuing response >Thread-81::DEBUG::2014-09-25 18:18:49,811::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:18:49,832::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00790012 s, 47.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:49,833::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:49,834::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:49,834::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:49,834::__init__::232::IOProcess::(_processLogs) (111068) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:49,842::__init__::232::IOProcess::(_processLogs) (111068) Queuing response >Dummy-148::DEBUG::2014-09-25 18:18:51,316::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:51,342::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163308 s, 62.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:51,477::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:51,479::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72124::DEBUG::2014-09-25 18:18:51,482::task::595::Storage.TaskManager.Task::(_updateState) Task=`5fac6732-4e40-4644-a1f2-ffbbda633d35`::moving from state init -> state preparing >Thread-72124::INFO::2014-09-25 18:18:51,482::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72124::INFO::2014-09-25 18:18:51,483::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00165142', 'lastCheck': '4.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00790012', 'lastCheck': '1.6', 'valid': True}} >Thread-72124::DEBUG::2014-09-25 18:18:51,483::task::1191::Storage.TaskManager.Task::(prepare) Task=`5fac6732-4e40-4644-a1f2-ffbbda633d35`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00165142', 'lastCheck': '4.9', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '3.3', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00790012', 'lastCheck': '1.6', 'valid': True}} >Thread-72124::DEBUG::2014-09-25 18:18:51,483::task::595::Storage.TaskManager.Task::(_updateState) Task=`5fac6732-4e40-4644-a1f2-ffbbda633d35`::moving from state preparing -> state finished >Thread-72124::DEBUG::2014-09-25 18:18:51,483::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72124::DEBUG::2014-09-25 18:18:51,484::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72124::DEBUG::2014-09-25 18:18:51,484::task::993::Storage.TaskManager.Task::(_decref) Task=`5fac6732-4e40-4644-a1f2-ffbbda633d35`::ref 0 aborting False >Thread-72124::DEBUG::2014-09-25 18:18:51,486::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:51,502::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:51,502::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72125::DEBUG::2014-09-25 18:18:51,504::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:52,808::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:52,809::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72126::DEBUG::2014-09-25 18:18:52,809::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72126::DEBUG::2014-09-25 18:18:52,811::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::moving from state init -> state preparing >Thread-72126::INFO::2014-09-25 18:18:52,811::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72126::ERROR::2014-09-25 18:18:52,812::task::866::Storage.TaskManager.Task::(_setError) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72126::DEBUG::2014-09-25 18:18:52,812::task::885::Storage.TaskManager.Task::(_run) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::Task._run: 3ebe7097-58ef-43a0-b98a-b7632be1e63a (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72126::DEBUG::2014-09-25 18:18:52,812::task::1217::Storage.TaskManager.Task::(stop) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::stopping in state preparing (force False) >Thread-72126::DEBUG::2014-09-25 18:18:52,813::task::993::Storage.TaskManager.Task::(_decref) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::ref 1 aborting True >Thread-72126::INFO::2014-09-25 18:18:52,813::task::1171::Storage.TaskManager.Task::(prepare) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::aborting: Task is aborted: u'' - code 100 >Thread-72126::DEBUG::2014-09-25 18:18:52,813::task::1176::Storage.TaskManager.Task::(prepare) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::Prepare: aborted: >Thread-72126::DEBUG::2014-09-25 18:18:52,813::task::993::Storage.TaskManager.Task::(_decref) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::ref 0 aborting True >Thread-72126::DEBUG::2014-09-25 18:18:52,813::task::928::Storage.TaskManager.Task::(_doAbort) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::Task._doAbort: force False >Thread-72126::DEBUG::2014-09-25 18:18:52,814::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72126::DEBUG::2014-09-25 18:18:52,814::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::moving from state preparing -> state aborting >Thread-72126::DEBUG::2014-09-25 18:18:52,814::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::_aborting: recover policy none >Thread-72126::DEBUG::2014-09-25 18:18:52,814::task::595::Storage.TaskManager.Task::(_updateState) Task=`3ebe7097-58ef-43a0-b98a-b7632be1e63a`::moving from state aborting -> state failed >Thread-72126::DEBUG::2014-09-25 18:18:52,814::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72126::DEBUG::2014-09-25 18:18:52,815::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72126::ERROR::2014-09-25 18:18:52,815::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72126::DEBUG::2014-09-25 18:18:52,815::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:53,362::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:53,391::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151144 s, 67.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:54,558::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:54,561::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72127::DEBUG::2014-09-25 18:18:54,566::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:18:55,411::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:55,446::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0200922 s, 51.0 MB/s\n'; <rc> = 0 >Thread-79::DEBUG::2014-09-25 18:18:56,565::domainMonitor::209::Storage.DomainMonitorThread::(_monitorDomain) Refreshing domain eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 18:18:56,570::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,570::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,570::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,571::__init__::232::IOProcess::(_processLogs) (111069) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:18:56,571::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,571::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,572::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,572::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,573::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,573::__init__::232::IOProcess::(_processLogs) (111070) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:18:56,574::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,574::__init__::232::IOProcess::(_processLogs) (111071) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:18:56,574::__init__::232::IOProcess::(_processLogs) (111069) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:56,580::__init__::232::IOProcess::(_processLogs) (111071) Queuing response >Thread-79::DEBUG::2014-09-25 18:18:56,584::fileSD::152::Storage.StorageDomain::(__init__) Reading domain in path /rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc >Thread-29::DEBUG::2014-09-25 18:18:56,585::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,585::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,585::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,586::__init__::232::IOProcess::(_processLogs) (111072) Got request for method 'touch' >Thread-29::DEBUG::2014-09-25 18:18:56,613::__init__::232::IOProcess::(_processLogs) (111072) Queuing response >Thread-79::DEBUG::2014-09-25 18:18:56,613::persistentDict::192::Storage.PersistentDict::(__init__) Created a persistent dict with FileMetadataRW backend >Thread-29::DEBUG::2014-09-25 18:18:56,615::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,615::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,616::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,616::__init__::232::IOProcess::(_processLogs) (111073) Got request for method 'readfile' >Thread-79::DEBUG::2014-09-25 18:18:56,620::persistentDict::234::Storage.PersistentDict::(refresh) read lines (FileMetadataRW)=['CLASS=Data', 'DESCRIPTION=master', 'IOOPTIMEOUTSEC=10', 'LEASERETRIES=3', 'LEASETIMESEC=60', 'LOCKPOLICY=', 'LOCKRENEWALINTERVALSEC=5', 'MASTER_VERSION=1', 'POOL_DESCRIPTION=Default', 'POOL_DOMAINS=eb633636-3019-47a9-9f5a-fc81f8dfd7dc:Active,fc587b6a-bac3-4668-99de-7f6fd6acce7f:Active,3573f707-d0c4-4dc9-b09f-dca9a3506a4f:Active', 'POOL_SPM_ID=2', 'POOL_SPM_LVER=7', 'POOL_UUID=00000002-0002-0002-0002-0000000001ad', 'REMOTE_PATH=10.34.63.202:/mnt/export/nfs/lv3/pbenas/35', 'ROLE=Master', 'SDUUID=eb633636-3019-47a9-9f5a-fc81f8dfd7dc', 'TYPE=NFS', 'VERSION=0', '_SHA_CKSUM=6a4641f693bfd2cc3c79303716ed9bccd9b33d8c'] >Thread-29::DEBUG::2014-09-25 18:18:56,620::__init__::232::IOProcess::(_processLogs) (111073) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:56,622::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,622::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,623::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,623::__init__::232::IOProcess::(_processLogs) (111074) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:56,624::__init__::232::IOProcess::(_processLogs) (111074) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:56,624::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,624::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,625::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,625::__init__::232::IOProcess::(_processLogs) (111075) Got request for method 'glob' >Thread-79::DEBUG::2014-09-25 18:18:56,633::fileSD::610::Storage.StorageDomain::(imageGarbageCollector) Removing remnants of deleted images [] >Thread-29::DEBUG::2014-09-25 18:18:56,633::__init__::232::IOProcess::(_processLogs) (111075) Queuing response >Thread-79::INFO::2014-09-25 18:18:56,634::sd::385::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_imageNS already registered >Thread-79::INFO::2014-09-25 18:18:56,635::sd::393::Storage.StorageDomain::(_registerResourceNamespaces) Resource namespace eb633636-3019-47a9-9f5a-fc81f8dfd7dc_volumeNS already registered >Thread-29::DEBUG::2014-09-25 18:18:56,636::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,636::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,636::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,637::__init__::232::IOProcess::(_processLogs) (111076) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:56,639::__init__::232::IOProcess::(_processLogs) (111076) Queuing response >Thread-79::DEBUG::2014-09-25 18:18:56,640::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:18:56,656::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00372108 s, 151 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:56,658::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,658::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,658::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,658::__init__::232::IOProcess::(_processLogs) (111077) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:56,662::__init__::232::IOProcess::(_processLogs) (111077) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:56,663::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,664::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,664::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,664::__init__::232::IOProcess::(_processLogs) (111078) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:56,667::__init__::232::IOProcess::(_processLogs) (111078) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:56,668::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,668::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,669::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,669::__init__::232::IOProcess::(_processLogs) (111079) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:56,670::__init__::232::IOProcess::(_processLogs) (111079) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:56,670::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:56,671::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:56,671::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:56,671::__init__::232::IOProcess::(_processLogs) (111080) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:18:56,676::__init__::232::IOProcess::(_processLogs) (111080) Queuing response >Dummy-148::DEBUG::2014-09-25 18:18:57,465::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:57,494::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0166509 s, 61.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:18:57,580::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:18:57,582::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72131::DEBUG::2014-09-25 18:18:57,585::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:18:58,175::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:58,176::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:58,176::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:58,176::__init__::232::IOProcess::(_processLogs) (111081) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:18:58,177::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:58,177::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:58,178::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:58,178::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:58,179::__init__::232::IOProcess::(_processLogs) (111081) Queuing response >Thread-29::DEBUG::2014-09-25 18:18:58,179::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:58,179::__init__::232::IOProcess::(_processLogs) (111082) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:18:58,179::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:58,179::__init__::232::IOProcess::(_processLogs) (111083) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:18:58,180::__init__::232::IOProcess::(_processLogs) (111083) Queuing response >Dummy-148::DEBUG::2014-09-25 18:18:59,513::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:18:59,600::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.071927 s, 14.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:59,844::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:59,845::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:59,845::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:59,845::__init__::232::IOProcess::(_processLogs) (111084) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:59,847::__init__::232::IOProcess::(_processLogs) (111084) Queuing response >Thread-81::DEBUG::2014-09-25 18:18:59,848::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:18:59,860::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00226865 s, 164 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:18:59,861::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:18:59,861::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:18:59,862::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:18:59,862::__init__::232::IOProcess::(_processLogs) (111085) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:18:59,863::__init__::232::IOProcess::(_processLogs) (111085) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:00,603::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:00,604::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72135::DEBUG::2014-09-25 18:19:00,606::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:01,619::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:01,642::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0142405 s, 71.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:02,840::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72136::DEBUG::2014-09-25 18:19:02,842::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:19:02,842::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72136::DEBUG::2014-09-25 18:19:02,844::task::595::Storage.TaskManager.Task::(_updateState) Task=`59900130-a855-402e-aede-be83f27cf452`::moving from state init -> state preparing >Thread-72136::INFO::2014-09-25 18:19:02,844::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72136::ERROR::2014-09-25 18:19:02,845::task::866::Storage.TaskManager.Task::(_setError) Task=`59900130-a855-402e-aede-be83f27cf452`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72136::DEBUG::2014-09-25 18:19:02,845::task::885::Storage.TaskManager.Task::(_run) Task=`59900130-a855-402e-aede-be83f27cf452`::Task._run: 59900130-a855-402e-aede-be83f27cf452 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72136::DEBUG::2014-09-25 18:19:02,846::task::1217::Storage.TaskManager.Task::(stop) Task=`59900130-a855-402e-aede-be83f27cf452`::stopping in state preparing (force False) >Thread-72136::DEBUG::2014-09-25 18:19:02,846::task::993::Storage.TaskManager.Task::(_decref) Task=`59900130-a855-402e-aede-be83f27cf452`::ref 1 aborting True >Thread-72136::INFO::2014-09-25 18:19:02,846::task::1171::Storage.TaskManager.Task::(prepare) Task=`59900130-a855-402e-aede-be83f27cf452`::aborting: Task is aborted: u'' - code 100 >Thread-72136::DEBUG::2014-09-25 18:19:02,846::task::1176::Storage.TaskManager.Task::(prepare) Task=`59900130-a855-402e-aede-be83f27cf452`::Prepare: aborted: >Thread-72136::DEBUG::2014-09-25 18:19:02,847::task::993::Storage.TaskManager.Task::(_decref) Task=`59900130-a855-402e-aede-be83f27cf452`::ref 0 aborting True >Thread-72136::DEBUG::2014-09-25 18:19:02,847::task::928::Storage.TaskManager.Task::(_doAbort) Task=`59900130-a855-402e-aede-be83f27cf452`::Task._doAbort: force False >Thread-72136::DEBUG::2014-09-25 18:19:02,847::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72136::DEBUG::2014-09-25 18:19:02,847::task::595::Storage.TaskManager.Task::(_updateState) Task=`59900130-a855-402e-aede-be83f27cf452`::moving from state preparing -> state aborting >Thread-72136::DEBUG::2014-09-25 18:19:02,847::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`59900130-a855-402e-aede-be83f27cf452`::_aborting: recover policy none >Thread-72136::DEBUG::2014-09-25 18:19:02,848::task::595::Storage.TaskManager.Task::(_updateState) Task=`59900130-a855-402e-aede-be83f27cf452`::moving from state aborting -> state failed >Thread-72136::DEBUG::2014-09-25 18:19:02,848::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72136::DEBUG::2014-09-25 18:19:02,848::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72136::ERROR::2014-09-25 18:19:02,848::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72136::DEBUG::2014-09-25 18:19:02,849::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:03,621::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:03,622::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72137::DEBUG::2014-09-25 18:19:03,624::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:03,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:03,690::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169779 s, 60.3 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:19:05,709::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:05,742::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0171554 s, 59.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:06,638::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:06,639::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72138::DEBUG::2014-09-25 18:19:06,642::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b296819-9b89-452d-bb0f-e4baeed13736`::moving from state init -> state preparing >Thread-72138::INFO::2014-09-25 18:19:06,642::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72138::INFO::2014-09-25 18:19:06,643::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00372108', 'lastCheck': '10.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00226865', 'lastCheck': '6.8', 'valid': True}} >Thread-72138::DEBUG::2014-09-25 18:19:06,643::task::1191::Storage.TaskManager.Task::(prepare) Task=`0b296819-9b89-452d-bb0f-e4baeed13736`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00372108', 'lastCheck': '10.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '18.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00226865', 'lastCheck': '6.8', 'valid': True}} >Thread-72138::DEBUG::2014-09-25 18:19:06,643::task::595::Storage.TaskManager.Task::(_updateState) Task=`0b296819-9b89-452d-bb0f-e4baeed13736`::moving from state preparing -> state finished >Thread-72138::DEBUG::2014-09-25 18:19:06,643::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72138::DEBUG::2014-09-25 18:19:06,644::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72138::DEBUG::2014-09-25 18:19:06,644::task::993::Storage.TaskManager.Task::(_decref) Task=`0b296819-9b89-452d-bb0f-e4baeed13736`::ref 0 aborting False >Thread-72138::DEBUG::2014-09-25 18:19:06,646::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:06,661::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:06,663::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72139::DEBUG::2014-09-25 18:19:06,664::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:19:06,677::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:06,678::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:06,678::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:06,678::__init__::232::IOProcess::(_processLogs) (111086) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:06,678::__init__::232::IOProcess::(_processLogs) (111086) Queuing response >Thread-79::DEBUG::2014-09-25 18:19:06,680::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:19:06,689::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000398344 s, 1.4 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:06,690::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:06,690::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:06,690::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:06,691::__init__::232::IOProcess::(_processLogs) (111087) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:06,691::__init__::232::IOProcess::(_processLogs) (111087) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:06,692::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:06,692::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:06,693::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:06,693::__init__::232::IOProcess::(_processLogs) (111088) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:06,694::__init__::232::IOProcess::(_processLogs) (111088) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:06,694::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:06,694::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:06,695::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:06,696::__init__::232::IOProcess::(_processLogs) (111089) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:06,696::__init__::232::IOProcess::(_processLogs) (111089) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:06,696::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:06,696::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:06,697::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:06,698::__init__::232::IOProcess::(_processLogs) (111090) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:06,698::__init__::232::IOProcess::(_processLogs) (111090) Queuing response >Dummy-148::DEBUG::2014-09-25 18:19:07,762::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:07,789::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158235 s, 64.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:09,711::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:09,712::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72140::DEBUG::2014-09-25 18:19:09,714::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:09,808::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:09,838::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0163416 s, 62.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:09,865::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:09,866::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:09,866::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:09,866::__init__::232::IOProcess::(_processLogs) (111091) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:09,882::__init__::232::IOProcess::(_processLogs) (111091) Queuing response >Thread-81::DEBUG::2014-09-25 18:19:09,882::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:19:09,903::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00279664 s, 133 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:09,904::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:09,905::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:09,905::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:09,905::__init__::232::IOProcess::(_processLogs) (111092) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:09,909::__init__::232::IOProcess::(_processLogs) (111092) Queuing response >Dummy-148::DEBUG::2014-09-25 18:19:11,858::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:11,883::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0158718 s, 64.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:12,729::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:12,731::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72141::DEBUG::2014-09-25 18:19:12,733::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:12,874::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72142::DEBUG::2014-09-25 18:19:12,876::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:19:12,876::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72142::DEBUG::2014-09-25 18:19:12,878::task::595::Storage.TaskManager.Task::(_updateState) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::moving from state init -> state preparing >Thread-72142::INFO::2014-09-25 18:19:12,878::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72142::ERROR::2014-09-25 18:19:12,879::task::866::Storage.TaskManager.Task::(_setError) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72142::DEBUG::2014-09-25 18:19:12,879::task::885::Storage.TaskManager.Task::(_run) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::Task._run: 9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72142::DEBUG::2014-09-25 18:19:12,879::task::1217::Storage.TaskManager.Task::(stop) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::stopping in state preparing (force False) >Thread-72142::DEBUG::2014-09-25 18:19:12,880::task::993::Storage.TaskManager.Task::(_decref) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::ref 1 aborting True >Thread-72142::INFO::2014-09-25 18:19:12,880::task::1171::Storage.TaskManager.Task::(prepare) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::aborting: Task is aborted: u'' - code 100 >Thread-72142::DEBUG::2014-09-25 18:19:12,880::task::1176::Storage.TaskManager.Task::(prepare) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::Prepare: aborted: >Thread-72142::DEBUG::2014-09-25 18:19:12,880::task::993::Storage.TaskManager.Task::(_decref) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::ref 0 aborting True >Thread-72142::DEBUG::2014-09-25 18:19:12,881::task::928::Storage.TaskManager.Task::(_doAbort) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::Task._doAbort: force False >Thread-72142::DEBUG::2014-09-25 18:19:12,881::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72142::DEBUG::2014-09-25 18:19:12,881::task::595::Storage.TaskManager.Task::(_updateState) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::moving from state preparing -> state aborting >Thread-72142::DEBUG::2014-09-25 18:19:12,881::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::_aborting: recover policy none >Thread-72142::DEBUG::2014-09-25 18:19:12,881::task::595::Storage.TaskManager.Task::(_updateState) Task=`9fb81be7-b8f1-4e3a-b3c7-3937c0b8742e`::moving from state aborting -> state failed >Thread-72142::DEBUG::2014-09-25 18:19:12,882::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72142::DEBUG::2014-09-25 18:19:12,882::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72142::ERROR::2014-09-25 18:19:12,882::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72142::DEBUG::2014-09-25 18:19:12,883::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:13,903::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:13,934::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0169461 s, 60.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:15,747::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:15,748::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72143::DEBUG::2014-09-25 18:19:15,751::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:15,954::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:15,986::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0207821 s, 49.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:16,699::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:16,700::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:16,701::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:16,702::__init__::232::IOProcess::(_processLogs) (111093) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:16,703::__init__::232::IOProcess::(_processLogs) (111093) Queuing response >Thread-79::DEBUG::2014-09-25 18:19:16,704::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:19:16,723::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00761705 s, 73.8 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:16,725::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:16,726::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:16,727::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:16,727::__init__::232::IOProcess::(_processLogs) (111094) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:16,729::__init__::232::IOProcess::(_processLogs) (111094) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:16,730::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:16,731::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:16,732::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:16,733::__init__::232::IOProcess::(_processLogs) (111095) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:16,734::__init__::232::IOProcess::(_processLogs) (111095) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:16,735::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:16,736::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:16,737::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:16,738::__init__::232::IOProcess::(_processLogs) (111096) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:16,739::__init__::232::IOProcess::(_processLogs) (111096) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:16,740::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:16,741::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:16,742::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:16,743::__init__::232::IOProcess::(_processLogs) (111097) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:16,744::__init__::232::IOProcess::(_processLogs) (111097) Queuing response >Dummy-148::DEBUG::2014-09-25 18:19:18,006::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:18,037::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159267 s, 64.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:18,769::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:18,771::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72144::DEBUG::2014-09-25 18:19:18,773::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:19:19,911::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:19,912::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:19,912::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:19,912::__init__::232::IOProcess::(_processLogs) (111098) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:19,917::__init__::232::IOProcess::(_processLogs) (111098) Queuing response >Thread-81::DEBUG::2014-09-25 18:19:19,918::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:19:19,930::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00241885 s, 153 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:19,931::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:19,932::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:19,932::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:19,932::__init__::232::IOProcess::(_processLogs) (111099) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:19,933::__init__::232::IOProcess::(_processLogs) (111099) Queuing response >Dummy-148::DEBUG::2014-09-25 18:19:20,057::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:20,079::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135206 s, 75.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:21,786::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:21,787::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72145::DEBUG::2014-09-25 18:19:21,790::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d779cc9-344e-4df5-b776-f14e596997ef`::moving from state init -> state preparing >Thread-72145::INFO::2014-09-25 18:19:21,790::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72145::INFO::2014-09-25 18:19:21,791::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00761705', 'lastCheck': '5.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00241885', 'lastCheck': '1.9', 'valid': True}} >Thread-72145::DEBUG::2014-09-25 18:19:21,791::task::1191::Storage.TaskManager.Task::(prepare) Task=`7d779cc9-344e-4df5-b776-f14e596997ef`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00761705', 'lastCheck': '5.0', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '33.6', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00241885', 'lastCheck': '1.9', 'valid': True}} >Thread-72145::DEBUG::2014-09-25 18:19:21,791::task::595::Storage.TaskManager.Task::(_updateState) Task=`7d779cc9-344e-4df5-b776-f14e596997ef`::moving from state preparing -> state finished >Thread-72145::DEBUG::2014-09-25 18:19:21,792::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72145::DEBUG::2014-09-25 18:19:21,792::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72145::DEBUG::2014-09-25 18:19:21,792::task::993::Storage.TaskManager.Task::(_decref) Task=`7d779cc9-344e-4df5-b776-f14e596997ef`::ref 0 aborting False >Thread-72145::DEBUG::2014-09-25 18:19:21,794::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:21,819::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:21,820::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72146::DEBUG::2014-09-25 18:19:21,822::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:22,099::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:22,128::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0173744 s, 58.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:22,906::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72147::DEBUG::2014-09-25 18:19:22,907::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:19:22,907::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72147::DEBUG::2014-09-25 18:19:22,910::task::595::Storage.TaskManager.Task::(_updateState) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::moving from state init -> state preparing >Thread-72147::INFO::2014-09-25 18:19:22,910::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72147::ERROR::2014-09-25 18:19:22,910::task::866::Storage.TaskManager.Task::(_setError) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72147::DEBUG::2014-09-25 18:19:22,911::task::885::Storage.TaskManager.Task::(_run) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::Task._run: c81dd0d3-4f59-404e-a029-6edca17bdcc8 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72147::DEBUG::2014-09-25 18:19:22,911::task::1217::Storage.TaskManager.Task::(stop) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::stopping in state preparing (force False) >Thread-72147::DEBUG::2014-09-25 18:19:22,911::task::993::Storage.TaskManager.Task::(_decref) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::ref 1 aborting True >Thread-72147::INFO::2014-09-25 18:19:22,912::task::1171::Storage.TaskManager.Task::(prepare) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::aborting: Task is aborted: u'' - code 100 >Thread-72147::DEBUG::2014-09-25 18:19:22,912::task::1176::Storage.TaskManager.Task::(prepare) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::Prepare: aborted: >Thread-72147::DEBUG::2014-09-25 18:19:22,912::task::993::Storage.TaskManager.Task::(_decref) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::ref 0 aborting True >Thread-72147::DEBUG::2014-09-25 18:19:22,912::task::928::Storage.TaskManager.Task::(_doAbort) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::Task._doAbort: force False >Thread-72147::DEBUG::2014-09-25 18:19:22,912::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72147::DEBUG::2014-09-25 18:19:22,913::task::595::Storage.TaskManager.Task::(_updateState) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::moving from state preparing -> state aborting >Thread-72147::DEBUG::2014-09-25 18:19:22,913::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::_aborting: recover policy none >Thread-72147::DEBUG::2014-09-25 18:19:22,913::task::595::Storage.TaskManager.Task::(_updateState) Task=`c81dd0d3-4f59-404e-a029-6edca17bdcc8`::moving from state aborting -> state failed >Thread-72147::DEBUG::2014-09-25 18:19:22,913::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72147::DEBUG::2014-09-25 18:19:22,914::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72147::ERROR::2014-09-25 18:19:22,914::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72147::DEBUG::2014-09-25 18:19:22,914::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:24,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:24,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164223 s, 62.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:24,863::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:24,864::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72148::DEBUG::2014-09-25 18:19:24,866::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:26,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:26,224::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0151037 s, 67.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:26,744::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:26,744::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:26,744::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:26,745::__init__::232::IOProcess::(_processLogs) (111100) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:26,754::__init__::232::IOProcess::(_processLogs) (111100) Queuing response >Thread-79::DEBUG::2014-09-25 18:19:26,755::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:19:26,773::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0017529 s, 321 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:26,774::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:26,774::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:26,775::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:26,775::__init__::232::IOProcess::(_processLogs) (111101) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:26,781::__init__::232::IOProcess::(_processLogs) (111101) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:26,783::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:26,783::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:26,784::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:26,784::__init__::232::IOProcess::(_processLogs) (111102) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:26,784::__init__::232::IOProcess::(_processLogs) (111102) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:26,785::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:26,785::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:26,786::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:26,786::__init__::232::IOProcess::(_processLogs) (111103) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:26,786::__init__::232::IOProcess::(_processLogs) (111103) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:26,787::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:26,787::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:26,788::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:26,789::__init__::232::IOProcess::(_processLogs) (111104) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:26,789::__init__::232::IOProcess::(_processLogs) (111104) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:27,884::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:27,886::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72149::DEBUG::2014-09-25 18:19:27,888::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:28,244::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:28,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0170227 s, 60.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:29,936::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:29,936::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:29,936::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:29,937::__init__::232::IOProcess::(_processLogs) (111105) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:29,937::__init__::232::IOProcess::(_processLogs) (111105) Queuing response >Thread-81::DEBUG::2014-09-25 18:19:29,938::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:19:29,947::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000369798 s, 1.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:29,949::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:29,949::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:29,949::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:29,949::__init__::232::IOProcess::(_processLogs) (111106) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:29,950::__init__::232::IOProcess::(_processLogs) (111106) Queuing response >Dummy-148::DEBUG::2014-09-25 18:19:30,292::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:30,319::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017675 s, 57.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:30,901::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:30,903::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72150::DEBUG::2014-09-25 18:19:30,905::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:32,339::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:32,361::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0130974 s, 78.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:32,938::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72151::DEBUG::2014-09-25 18:19:32,940::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:19:32,940::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72151::DEBUG::2014-09-25 18:19:32,942::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::moving from state init -> state preparing >Thread-72151::INFO::2014-09-25 18:19:32,943::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72151::ERROR::2014-09-25 18:19:32,943::task::866::Storage.TaskManager.Task::(_setError) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72151::DEBUG::2014-09-25 18:19:32,944::task::885::Storage.TaskManager.Task::(_run) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::Task._run: 5d2a51d1-4922-45ce-a509-a2487a67a065 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72151::DEBUG::2014-09-25 18:19:32,944::task::1217::Storage.TaskManager.Task::(stop) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::stopping in state preparing (force False) >Thread-72151::DEBUG::2014-09-25 18:19:32,944::task::993::Storage.TaskManager.Task::(_decref) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::ref 1 aborting True >Thread-72151::INFO::2014-09-25 18:19:32,944::task::1171::Storage.TaskManager.Task::(prepare) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::aborting: Task is aborted: u'' - code 100 >Thread-72151::DEBUG::2014-09-25 18:19:32,945::task::1176::Storage.TaskManager.Task::(prepare) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::Prepare: aborted: >Thread-72151::DEBUG::2014-09-25 18:19:32,945::task::993::Storage.TaskManager.Task::(_decref) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::ref 0 aborting True >Thread-72151::DEBUG::2014-09-25 18:19:32,945::task::928::Storage.TaskManager.Task::(_doAbort) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::Task._doAbort: force False >Thread-72151::DEBUG::2014-09-25 18:19:32,945::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72151::DEBUG::2014-09-25 18:19:32,945::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::moving from state preparing -> state aborting >Thread-72151::DEBUG::2014-09-25 18:19:32,946::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::_aborting: recover policy none >Thread-72151::DEBUG::2014-09-25 18:19:32,946::task::595::Storage.TaskManager.Task::(_updateState) Task=`5d2a51d1-4922-45ce-a509-a2487a67a065`::moving from state aborting -> state failed >Thread-72151::DEBUG::2014-09-25 18:19:32,946::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72151::DEBUG::2014-09-25 18:19:32,946::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72151::ERROR::2014-09-25 18:19:32,947::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72151::DEBUG::2014-09-25 18:19:32,947::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:33,917::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:33,918::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72152::DEBUG::2014-09-25 18:19:33,920::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:34,380::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:34,408::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0144939 s, 70.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:19:36,428::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:36,454::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013973 s, 73.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:36,790::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:36,790::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:36,790::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:36,790::__init__::232::IOProcess::(_processLogs) (111107) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:36,794::__init__::232::IOProcess::(_processLogs) (111107) Queuing response >Thread-79::DEBUG::2014-09-25 18:19:36,795::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:19:36,813::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00306692 s, 183 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:36,814::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:36,815::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:36,815::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:36,815::__init__::232::IOProcess::(_processLogs) (111108) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:36,818::__init__::232::IOProcess::(_processLogs) (111108) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:36,819::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:36,820::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:36,820::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:36,821::__init__::232::IOProcess::(_processLogs) (111109) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:36,821::__init__::232::IOProcess::(_processLogs) (111109) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:36,821::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:36,822::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:36,822::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:36,823::__init__::232::IOProcess::(_processLogs) (111110) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:36,823::__init__::232::IOProcess::(_processLogs) (111110) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:36,824::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:36,824::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:36,824::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:36,825::__init__::232::IOProcess::(_processLogs) (111111) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:36,826::__init__::232::IOProcess::(_processLogs) (111111) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:36,932::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:36,934::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72153::DEBUG::2014-09-25 18:19:36,937::task::595::Storage.TaskManager.Task::(_updateState) Task=`08f714c0-0565-4bcb-a76e-1d38cef7d5c2`::moving from state init -> state preparing >Thread-72153::INFO::2014-09-25 18:19:36,937::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72153::INFO::2014-09-25 18:19:36,937::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00306692', 'lastCheck': '0.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.8', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000369798', 'lastCheck': '7.0', 'valid': True}} >Thread-72153::DEBUG::2014-09-25 18:19:36,938::task::1191::Storage.TaskManager.Task::(prepare) Task=`08f714c0-0565-4bcb-a76e-1d38cef7d5c2`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00306692', 'lastCheck': '0.1', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '48.8', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.000369798', 'lastCheck': '7.0', 'valid': True}} >Thread-72153::DEBUG::2014-09-25 18:19:36,938::task::595::Storage.TaskManager.Task::(_updateState) Task=`08f714c0-0565-4bcb-a76e-1d38cef7d5c2`::moving from state preparing -> state finished >Thread-72153::DEBUG::2014-09-25 18:19:36,938::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72153::DEBUG::2014-09-25 18:19:36,938::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72153::DEBUG::2014-09-25 18:19:36,939::task::993::Storage.TaskManager.Task::(_decref) Task=`08f714c0-0565-4bcb-a76e-1d38cef7d5c2`::ref 0 aborting False >Thread-72153::DEBUG::2014-09-25 18:19:36,941::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:36,957::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:36,958::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72154::DEBUG::2014-09-25 18:19:36,960::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:38,474::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:38,500::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164933 s, 62.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:39,952::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:39,953::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:39,953::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:39,953::__init__::232::IOProcess::(_processLogs) (111112) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:39,961::__init__::232::IOProcess::(_processLogs) (111112) Queuing response >Thread-81::DEBUG::2014-09-25 18:19:39,962::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:19:39,976::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00293571 s, 126 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:39,977::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:39,977::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:39,978::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:39,978::__init__::232::IOProcess::(_processLogs) (111113) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:39,980::__init__::232::IOProcess::(_processLogs) (111113) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:40,021::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:40,022::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72155::DEBUG::2014-09-25 18:19:40,024::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:40,520::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:40,542::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0133323 s, 76.8 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:19:42,562::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:42,588::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0146017 s, 70.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:42,973::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:42,975::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72156::DEBUG::2014-09-25 18:19:42,975::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72156::DEBUG::2014-09-25 18:19:42,977::task::595::Storage.TaskManager.Task::(_updateState) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::moving from state init -> state preparing >Thread-72156::INFO::2014-09-25 18:19:42,977::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72156::ERROR::2014-09-25 18:19:42,978::task::866::Storage.TaskManager.Task::(_setError) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72156::DEBUG::2014-09-25 18:19:42,978::task::885::Storage.TaskManager.Task::(_run) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::Task._run: 6ff68e16-1b66-4f20-ac40-83f5ea63ea5c (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72156::DEBUG::2014-09-25 18:19:42,978::task::1217::Storage.TaskManager.Task::(stop) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::stopping in state preparing (force False) >Thread-72156::DEBUG::2014-09-25 18:19:42,979::task::993::Storage.TaskManager.Task::(_decref) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::ref 1 aborting True >Thread-72156::INFO::2014-09-25 18:19:42,979::task::1171::Storage.TaskManager.Task::(prepare) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::aborting: Task is aborted: u'' - code 100 >Thread-72156::DEBUG::2014-09-25 18:19:42,979::task::1176::Storage.TaskManager.Task::(prepare) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::Prepare: aborted: >Thread-72156::DEBUG::2014-09-25 18:19:42,979::task::993::Storage.TaskManager.Task::(_decref) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::ref 0 aborting True >Thread-72156::DEBUG::2014-09-25 18:19:42,980::task::928::Storage.TaskManager.Task::(_doAbort) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::Task._doAbort: force False >Thread-72156::DEBUG::2014-09-25 18:19:42,980::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72156::DEBUG::2014-09-25 18:19:42,980::task::595::Storage.TaskManager.Task::(_updateState) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::moving from state preparing -> state aborting >Thread-72156::DEBUG::2014-09-25 18:19:42,980::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::_aborting: recover policy none >Thread-72156::DEBUG::2014-09-25 18:19:42,980::task::595::Storage.TaskManager.Task::(_updateState) Task=`6ff68e16-1b66-4f20-ac40-83f5ea63ea5c`::moving from state aborting -> state failed >Thread-72156::DEBUG::2014-09-25 18:19:42,981::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72156::DEBUG::2014-09-25 18:19:42,981::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72156::ERROR::2014-09-25 18:19:42,981::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72156::DEBUG::2014-09-25 18:19:42,981::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:43,036::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:43,037::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72157::DEBUG::2014-09-25 18:19:43,039::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:44,607::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:44,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0220363 s, 46.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:46,052::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:46,055::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72158::DEBUG::2014-09-25 18:19:46,061::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:46,661::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:46,734::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0297124 s, 34.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:46,828::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:46,830::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:46,831::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:46,833::__init__::232::IOProcess::(_processLogs) (111114) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:46,836::__init__::232::IOProcess::(_processLogs) (111114) Queuing response >Thread-79::DEBUG::2014-09-25 18:19:46,838::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:19:46,859::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00531873 s, 106 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:46,863::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:46,865::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:46,867::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:46,869::__init__::232::IOProcess::(_processLogs) (111115) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:46,872::__init__::232::IOProcess::(_processLogs) (111115) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:46,876::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:46,878::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:46,879::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:46,881::__init__::232::IOProcess::(_processLogs) (111116) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:46,883::__init__::232::IOProcess::(_processLogs) (111116) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:46,887::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:46,888::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:46,890::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:46,892::__init__::232::IOProcess::(_processLogs) (111117) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:46,893::__init__::232::IOProcess::(_processLogs) (111117) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:46,897::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:46,899::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:46,900::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:46,902::__init__::232::IOProcess::(_processLogs) (111118) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:46,903::__init__::232::IOProcess::(_processLogs) (111118) Queuing response >Dummy-148::DEBUG::2014-09-25 18:19:48,949::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:48,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0214941 s, 47.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:49,082::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:49,083::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72159::DEBUG::2014-09-25 18:19:49,085::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:19:49,982::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:49,983::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:49,983::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:49,983::__init__::232::IOProcess::(_processLogs) (111119) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:49,985::__init__::232::IOProcess::(_processLogs) (111119) Queuing response >Thread-81::DEBUG::2014-09-25 18:19:49,986::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:19:50,002::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00251356 s, 148 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:50,003::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:50,003::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:50,004::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:50,004::__init__::232::IOProcess::(_processLogs) (111120) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:50,008::__init__::232::IOProcess::(_processLogs) (111120) Queuing response >Dummy-148::DEBUG::2014-09-25 18:19:51,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:51,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0135874 s, 75.4 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:52,097::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:52,098::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72160::DEBUG::2014-09-25 18:19:52,101::task::595::Storage.TaskManager.Task::(_updateState) Task=`222fc939-d522-404d-9baa-0a18c7898eed`::moving from state init -> state preparing >Thread-72160::INFO::2014-09-25 18:19:52,101::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72160::INFO::2014-09-25 18:19:52,102::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00531873', 'lastCheck': '5.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00251356', 'lastCheck': '2.1', 'valid': True}} >Thread-72160::DEBUG::2014-09-25 18:19:52,102::task::1191::Storage.TaskManager.Task::(prepare) Task=`222fc939-d522-404d-9baa-0a18c7898eed`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00531873', 'lastCheck': '5.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '63.9', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00251356', 'lastCheck': '2.1', 'valid': True}} >Thread-72160::DEBUG::2014-09-25 18:19:52,102::task::595::Storage.TaskManager.Task::(_updateState) Task=`222fc939-d522-404d-9baa-0a18c7898eed`::moving from state preparing -> state finished >Thread-72160::DEBUG::2014-09-25 18:19:52,102::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72160::DEBUG::2014-09-25 18:19:52,103::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72160::DEBUG::2014-09-25 18:19:52,103::task::993::Storage.TaskManager.Task::(_decref) Task=`222fc939-d522-404d-9baa-0a18c7898eed`::ref 0 aborting False >Thread-72160::DEBUG::2014-09-25 18:19:52,105::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:52,118::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:52,120::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72161::DEBUG::2014-09-25 18:19:52,125::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:53,005::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72162::DEBUG::2014-09-25 18:19:53,006::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:19:53,006::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72162::DEBUG::2014-09-25 18:19:53,009::task::595::Storage.TaskManager.Task::(_updateState) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::moving from state init -> state preparing >Thread-72162::INFO::2014-09-25 18:19:53,010::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72162::ERROR::2014-09-25 18:19:53,010::task::866::Storage.TaskManager.Task::(_setError) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72162::DEBUG::2014-09-25 18:19:53,011::task::885::Storage.TaskManager.Task::(_run) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::Task._run: 660350f5-a2f3-4dec-97eb-d621489f8dd7 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72162::DEBUG::2014-09-25 18:19:53,011::task::1217::Storage.TaskManager.Task::(stop) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::stopping in state preparing (force False) >Thread-72162::DEBUG::2014-09-25 18:19:53,011::task::993::Storage.TaskManager.Task::(_decref) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::ref 1 aborting True >Thread-72162::INFO::2014-09-25 18:19:53,012::task::1171::Storage.TaskManager.Task::(prepare) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::aborting: Task is aborted: u'' - code 100 >Thread-72162::DEBUG::2014-09-25 18:19:53,012::task::1176::Storage.TaskManager.Task::(prepare) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::Prepare: aborted: >Thread-72162::DEBUG::2014-09-25 18:19:53,012::task::993::Storage.TaskManager.Task::(_decref) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::ref 0 aborting True >Thread-72162::DEBUG::2014-09-25 18:19:53,012::task::928::Storage.TaskManager.Task::(_doAbort) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::Task._doAbort: force False >Thread-72162::DEBUG::2014-09-25 18:19:53,012::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72162::DEBUG::2014-09-25 18:19:53,013::task::595::Storage.TaskManager.Task::(_updateState) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::moving from state preparing -> state aborting >Thread-72162::DEBUG::2014-09-25 18:19:53,013::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::_aborting: recover policy none >Thread-72162::DEBUG::2014-09-25 18:19:53,013::task::595::Storage.TaskManager.Task::(_updateState) Task=`660350f5-a2f3-4dec-97eb-d621489f8dd7`::moving from state aborting -> state failed >Thread-72162::DEBUG::2014-09-25 18:19:53,013::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72162::DEBUG::2014-09-25 18:19:53,014::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72162::ERROR::2014-09-25 18:19:53,014::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72162::DEBUG::2014-09-25 18:19:53,014::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:53,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:53,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0179707 s, 57.0 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:19:55,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:55,131::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0182284 s, 56.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:55,164::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:55,165::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72163::DEBUG::2014-09-25 18:19:55,168::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-72129::WARNING::2014-09-25 18:19:56,569::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-29::DEBUG::2014-09-25 18:19:56,907::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:56,908::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:56,908::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:56,908::__init__::232::IOProcess::(_processLogs) (111121) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:56,913::__init__::232::IOProcess::(_processLogs) (111121) Queuing response >Thread-79::DEBUG::2014-09-25 18:19:56,915::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:19:56,930::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0037212 s, 151 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:19:56,931::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:56,932::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:56,932::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:56,932::__init__::232::IOProcess::(_processLogs) (111122) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:19:56,939::__init__::232::IOProcess::(_processLogs) (111122) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:56,940::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:56,940::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:56,941::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:56,941::__init__::232::IOProcess::(_processLogs) (111123) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:56,942::__init__::232::IOProcess::(_processLogs) (111123) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:56,943::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:56,943::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:56,943::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:56,943::__init__::232::IOProcess::(_processLogs) (111124) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:56,947::__init__::232::IOProcess::(_processLogs) (111124) Queuing response >Thread-29::DEBUG::2014-09-25 18:19:56,948::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:19:56,948::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:19:56,949::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:19:56,949::__init__::232::IOProcess::(_processLogs) (111125) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:19:56,951::__init__::232::IOProcess::(_processLogs) (111125) Queuing response >Dummy-148::DEBUG::2014-09-25 18:19:57,150::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:57,176::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0168396 s, 60.8 MB/s\n'; <rc> = 0 >Thread-72133::WARNING::2014-09-25 18:19:58,174::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:19:58,175::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:19:58,176::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:19:58,187::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:19:58,188::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72164::DEBUG::2014-09-25 18:19:58,190::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:19:59,196::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:19:59,236::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0238379 s, 43.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:00,010::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:00,011::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:00,011::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:00,011::__init__::232::IOProcess::(_processLogs) (111126) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:00,012::__init__::232::IOProcess::(_processLogs) (111126) Queuing response >Thread-81::DEBUG::2014-09-25 18:20:00,013::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:20:00,025::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00202772 s, 183 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:00,026::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:00,027::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:00,027::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:00,027::__init__::232::IOProcess::(_processLogs) (111127) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:00,029::__init__::232::IOProcess::(_processLogs) (111127) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:01,207::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:01,209::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72165::DEBUG::2014-09-25 18:20:01,211::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:01,256::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:01,294::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0216881 s, 47.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:03,038::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:03,039::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72166::DEBUG::2014-09-25 18:20:03,040::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72166::DEBUG::2014-09-25 18:20:03,042::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::moving from state init -> state preparing >Thread-72166::INFO::2014-09-25 18:20:03,042::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72166::ERROR::2014-09-25 18:20:03,042::task::866::Storage.TaskManager.Task::(_setError) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72166::DEBUG::2014-09-25 18:20:03,043::task::885::Storage.TaskManager.Task::(_run) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::Task._run: e3496e2c-7d42-4085-bc55-08fe56fe06cf (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72166::DEBUG::2014-09-25 18:20:03,043::task::1217::Storage.TaskManager.Task::(stop) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::stopping in state preparing (force False) >Thread-72166::DEBUG::2014-09-25 18:20:03,043::task::993::Storage.TaskManager.Task::(_decref) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::ref 1 aborting True >Thread-72166::INFO::2014-09-25 18:20:03,044::task::1171::Storage.TaskManager.Task::(prepare) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::aborting: Task is aborted: u'' - code 100 >Thread-72166::DEBUG::2014-09-25 18:20:03,044::task::1176::Storage.TaskManager.Task::(prepare) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::Prepare: aborted: >Thread-72166::DEBUG::2014-09-25 18:20:03,044::task::993::Storage.TaskManager.Task::(_decref) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::ref 0 aborting True >Thread-72166::DEBUG::2014-09-25 18:20:03,044::task::928::Storage.TaskManager.Task::(_doAbort) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::Task._doAbort: force False >Thread-72166::DEBUG::2014-09-25 18:20:03,044::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72166::DEBUG::2014-09-25 18:20:03,045::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::moving from state preparing -> state aborting >Thread-72166::DEBUG::2014-09-25 18:20:03,045::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::_aborting: recover policy none >Thread-72166::DEBUG::2014-09-25 18:20:03,045::task::595::Storage.TaskManager.Task::(_updateState) Task=`e3496e2c-7d42-4085-bc55-08fe56fe06cf`::moving from state aborting -> state failed >Thread-72166::DEBUG::2014-09-25 18:20:03,045::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72166::DEBUG::2014-09-25 18:20:03,045::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72166::ERROR::2014-09-25 18:20:03,046::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72166::DEBUG::2014-09-25 18:20:03,046::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:03,318::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:03,340::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134673 s, 76.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:04,224::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:04,225::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72167::DEBUG::2014-09-25 18:20:04,227::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:05,155::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72168::DEBUG::2014-09-25 18:20:05,156::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.connectStorageServer' in bridge with {u'connectionParams': [{u'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8', u'connection': u'pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', u'iqn': u'', u'user': u'', u'tpgt': u'', u'password': u'', u'port': u''}], u'storagepoolID': u'00000000-0000-0000-0000-000000000000', u'domainType': 1} >JsonRpcServer::DEBUG::2014-09-25 18:20:05,156::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72168::DEBUG::2014-09-25 18:20:05,158::task::595::Storage.TaskManager.Task::(_updateState) Task=`391a2aef-b5dc-420b-b2c9-2a709f5bf90c`::moving from state init -> state preparing >Thread-72168::INFO::2014-09-25 18:20:05,159::logUtils::44::dispatcher::(wrapper) Run and protect: connectStorageServer(domType=1, spUUID=u'00000000-0000-0000-0000-000000000000', conList=[{u'connection': u'pb-rh35.rhev.lab.eng.brq.redhat.com:/var/lib/exports/iso', u'iqn': u'', u'user': u'', u'tpgt': u'', u'password': '******', u'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8', u'port': u''}], options=None) >Thread-72168::DEBUG::2014-09-25 18:20:05,162::hsm::2372::Storage.HSM::(__prefetchDomains) nfs local path: /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Thread-29::DEBUG::2014-09-25 18:20:05,163::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:05,163::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:05,164::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:05,164::__init__::232::IOProcess::(_processLogs) (111128) Got request for method 'glob' >Dummy-148::DEBUG::2014-09-25 18:20:05,360::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:05,382::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013265 s, 77.2 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:06,953::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:06,953::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:06,953::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:06,953::__init__::232::IOProcess::(_processLogs) (111129) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:06,960::__init__::232::IOProcess::(_processLogs) (111129) Queuing response >Thread-79::DEBUG::2014-09-25 18:20:06,961::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:20:07,002::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.0175653 s, 32.0 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:07,003::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:07,004::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:07,004::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:07,004::__init__::232::IOProcess::(_processLogs) (111130) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:07,012::__init__::232::IOProcess::(_processLogs) (111130) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:07,013::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:07,014::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:07,014::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:07,015::__init__::232::IOProcess::(_processLogs) (111131) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:07,015::__init__::232::IOProcess::(_processLogs) (111131) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:07,015::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:07,015::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:07,016::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:07,017::__init__::232::IOProcess::(_processLogs) (111132) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:07,017::__init__::232::IOProcess::(_processLogs) (111132) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:07,017::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:07,018::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:07,019::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:07,019::__init__::232::IOProcess::(_processLogs) (111133) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:07,019::__init__::232::IOProcess::(_processLogs) (111133) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:07,239::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:07,240::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72169::DEBUG::2014-09-25 18:20:07,243::task::595::Storage.TaskManager.Task::(_updateState) Task=`1359b382-f1d2-4c6f-89a8-1f4af7b4cefb`::moving from state init -> state preparing >Thread-72169::INFO::2014-09-25 18:20:07,243::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72169::INFO::2014-09-25 18:20:07,244::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0175653', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.1', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00202772', 'lastCheck': '7.2', 'valid': True}} >Thread-72169::DEBUG::2014-09-25 18:20:07,244::task::1191::Storage.TaskManager.Task::(prepare) Task=`1359b382-f1d2-4c6f-89a8-1f4af7b4cefb`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0175653', 'lastCheck': '0.2', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '9.1', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00202772', 'lastCheck': '7.2', 'valid': True}} >Thread-72169::DEBUG::2014-09-25 18:20:07,244::task::595::Storage.TaskManager.Task::(_updateState) Task=`1359b382-f1d2-4c6f-89a8-1f4af7b4cefb`::moving from state preparing -> state finished >Thread-72169::DEBUG::2014-09-25 18:20:07,244::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72169::DEBUG::2014-09-25 18:20:07,245::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72169::DEBUG::2014-09-25 18:20:07,245::task::993::Storage.TaskManager.Task::(_decref) Task=`1359b382-f1d2-4c6f-89a8-1f4af7b4cefb`::ref 0 aborting False >Thread-72169::DEBUG::2014-09-25 18:20:07,247::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:07,262::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:07,263::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72170::DEBUG::2014-09-25 18:20:07,265::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:07,402::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:07,434::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184414 s, 55.5 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:08,181::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:08,182::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:08,182::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:08,182::__init__::232::IOProcess::(_processLogs) (111134) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:20:08,185::__init__::232::IOProcess::(_processLogs) (111134) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:08,187::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:08,188::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:08,189::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:08,190::__init__::232::IOProcess::(_processLogs) (111135) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:20:08,191::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:08,192::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:08,193::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:08,194::__init__::232::IOProcess::(_processLogs) (111136) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:20:08,198::__init__::232::IOProcess::(_processLogs) (111136) Queuing response >Dummy-148::DEBUG::2014-09-25 18:20:09,453::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:09,483::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.015593 s, 65.7 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:10,032::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:10,032::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:10,032::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:10,032::__init__::232::IOProcess::(_processLogs) (111137) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:10,039::__init__::232::IOProcess::(_processLogs) (111137) Queuing response >Thread-81::DEBUG::2014-09-25 18:20:10,039::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:20:10,061::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00213513 s, 174 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:10,062::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:10,063::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:10,063::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:10,063::__init__::232::IOProcess::(_processLogs) (111138) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:10,064::__init__::232::IOProcess::(_processLogs) (111138) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:10,315::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:10,316::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72174::DEBUG::2014-09-25 18:20:10,318::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:11,503::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:11,526::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0139056 s, 73.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:13,082::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:13,084::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72175::DEBUG::2014-09-25 18:20:13,084::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72175::DEBUG::2014-09-25 18:20:13,085::task::595::Storage.TaskManager.Task::(_updateState) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::moving from state init -> state preparing >Thread-72175::INFO::2014-09-25 18:20:13,086::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72175::ERROR::2014-09-25 18:20:13,086::task::866::Storage.TaskManager.Task::(_setError) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72175::DEBUG::2014-09-25 18:20:13,087::task::885::Storage.TaskManager.Task::(_run) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::Task._run: 6603f886-fefb-4336-ac40-5796f51213b4 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72175::DEBUG::2014-09-25 18:20:13,087::task::1217::Storage.TaskManager.Task::(stop) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::stopping in state preparing (force False) >Thread-72175::DEBUG::2014-09-25 18:20:13,087::task::993::Storage.TaskManager.Task::(_decref) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::ref 1 aborting True >Thread-72175::INFO::2014-09-25 18:20:13,087::task::1171::Storage.TaskManager.Task::(prepare) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::aborting: Task is aborted: u'' - code 100 >Thread-72175::DEBUG::2014-09-25 18:20:13,088::task::1176::Storage.TaskManager.Task::(prepare) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::Prepare: aborted: >Thread-72175::DEBUG::2014-09-25 18:20:13,088::task::993::Storage.TaskManager.Task::(_decref) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::ref 0 aborting True >Thread-72175::DEBUG::2014-09-25 18:20:13,088::task::928::Storage.TaskManager.Task::(_doAbort) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::Task._doAbort: force False >Thread-72175::DEBUG::2014-09-25 18:20:13,088::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72175::DEBUG::2014-09-25 18:20:13,089::task::595::Storage.TaskManager.Task::(_updateState) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::moving from state preparing -> state aborting >Thread-72175::DEBUG::2014-09-25 18:20:13,089::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::_aborting: recover policy none >Thread-72175::DEBUG::2014-09-25 18:20:13,089::task::595::Storage.TaskManager.Task::(_updateState) Task=`6603f886-fefb-4336-ac40-5796f51213b4`::moving from state aborting -> state failed >Thread-72175::DEBUG::2014-09-25 18:20:13,089::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72175::DEBUG::2014-09-25 18:20:13,090::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72175::ERROR::2014-09-25 18:20:13,090::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72175::DEBUG::2014-09-25 18:20:13,090::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:13,331::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:13,331::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72176::DEBUG::2014-09-25 18:20:13,333::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:13,545::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:13,575::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174314 s, 58.7 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:20:15,595::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:15,626::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.018795 s, 54.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:16,353::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:16,356::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72177::DEBUG::2014-09-25 18:20:16,362::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:20:17,019::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:17,020::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:17,020::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:17,020::__init__::232::IOProcess::(_processLogs) (111139) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:17,023::__init__::232::IOProcess::(_processLogs) (111139) Queuing response >Thread-79::DEBUG::2014-09-25 18:20:17,023::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:20:17,040::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00606749 s, 92.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:17,041::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:17,042::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:17,042::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:17,042::__init__::232::IOProcess::(_processLogs) (111140) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:17,045::__init__::232::IOProcess::(_processLogs) (111140) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:17,046::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:17,046::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:17,047::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:17,047::__init__::232::IOProcess::(_processLogs) (111141) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:17,047::__init__::232::IOProcess::(_processLogs) (111141) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:17,048::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:17,048::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:17,048::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:17,048::__init__::232::IOProcess::(_processLogs) (111142) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:17,048::__init__::232::IOProcess::(_processLogs) (111142) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:17,049::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:17,049::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:17,049::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:17,049::__init__::232::IOProcess::(_processLogs) (111143) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:17,049::__init__::232::IOProcess::(_processLogs) (111143) Queuing response >Dummy-148::DEBUG::2014-09-25 18:20:17,646::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:17,876::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.219254 s, 4.7 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:19,377::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:19,379::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72178::DEBUG::2014-09-25 18:20:19,381::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:19,896::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:19,931::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0237522 s, 43.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:20,066::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:20,066::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:20,067::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:20,067::__init__::232::IOProcess::(_processLogs) (111144) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:20,072::__init__::232::IOProcess::(_processLogs) (111144) Queuing response >Thread-81::DEBUG::2014-09-25 18:20:20,073::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:20:20,092::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00466861 s, 79.5 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:20,093::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:20,093::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:20,093::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:20,094::__init__::232::IOProcess::(_processLogs) (111145) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:20,098::__init__::232::IOProcess::(_processLogs) (111145) Queuing response >Dummy-148::DEBUG::2014-09-25 18:20:21,951::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:21,978::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0156764 s, 65.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:22,391::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:22,392::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72179::DEBUG::2014-09-25 18:20:22,395::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6b4e216-1926-48ad-ac31-e7a81e7588f9`::moving from state init -> state preparing >Thread-72179::INFO::2014-09-25 18:20:22,395::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72179::INFO::2014-09-25 18:20:22,396::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00606749', 'lastCheck': '5.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '24.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00466861', 'lastCheck': '2.3', 'valid': True}} >Thread-72179::DEBUG::2014-09-25 18:20:22,396::task::1191::Storage.TaskManager.Task::(prepare) Task=`f6b4e216-1926-48ad-ac31-e7a81e7588f9`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00606749', 'lastCheck': '5.3', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '24.2', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00466861', 'lastCheck': '2.3', 'valid': True}} >Thread-72179::DEBUG::2014-09-25 18:20:22,396::task::595::Storage.TaskManager.Task::(_updateState) Task=`f6b4e216-1926-48ad-ac31-e7a81e7588f9`::moving from state preparing -> state finished >Thread-72179::DEBUG::2014-09-25 18:20:22,397::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72179::DEBUG::2014-09-25 18:20:22,397::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72179::DEBUG::2014-09-25 18:20:22,397::task::993::Storage.TaskManager.Task::(_decref) Task=`f6b4e216-1926-48ad-ac31-e7a81e7588f9`::ref 0 aborting False >Thread-72179::DEBUG::2014-09-25 18:20:22,399::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:22,413::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:22,414::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72180::DEBUG::2014-09-25 18:20:22,416::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:23,118::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72181::DEBUG::2014-09-25 18:20:23,119::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:20:23,119::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72181::DEBUG::2014-09-25 18:20:23,121::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::moving from state init -> state preparing >Thread-72181::INFO::2014-09-25 18:20:23,121::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72181::ERROR::2014-09-25 18:20:23,122::task::866::Storage.TaskManager.Task::(_setError) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72181::DEBUG::2014-09-25 18:20:23,122::task::885::Storage.TaskManager.Task::(_run) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::Task._run: 3c32e056-d465-4c96-9824-4afbb4a55b5c (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72181::DEBUG::2014-09-25 18:20:23,122::task::1217::Storage.TaskManager.Task::(stop) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::stopping in state preparing (force False) >Thread-72181::DEBUG::2014-09-25 18:20:23,123::task::993::Storage.TaskManager.Task::(_decref) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::ref 1 aborting True >Thread-72181::INFO::2014-09-25 18:20:23,123::task::1171::Storage.TaskManager.Task::(prepare) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::aborting: Task is aborted: u'' - code 100 >Thread-72181::DEBUG::2014-09-25 18:20:23,123::task::1176::Storage.TaskManager.Task::(prepare) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::Prepare: aborted: >Thread-72181::DEBUG::2014-09-25 18:20:23,123::task::993::Storage.TaskManager.Task::(_decref) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::ref 0 aborting True >Thread-72181::DEBUG::2014-09-25 18:20:23,124::task::928::Storage.TaskManager.Task::(_doAbort) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::Task._doAbort: force False >Thread-72181::DEBUG::2014-09-25 18:20:23,124::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72181::DEBUG::2014-09-25 18:20:23,124::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::moving from state preparing -> state aborting >Thread-72181::DEBUG::2014-09-25 18:20:23,124::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::_aborting: recover policy none >Thread-72181::DEBUG::2014-09-25 18:20:23,124::task::595::Storage.TaskManager.Task::(_updateState) Task=`3c32e056-d465-4c96-9824-4afbb4a55b5c`::moving from state aborting -> state failed >Thread-72181::DEBUG::2014-09-25 18:20:23,125::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72181::DEBUG::2014-09-25 18:20:23,125::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72181::ERROR::2014-09-25 18:20:23,125::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72181::DEBUG::2014-09-25 18:20:23,125::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:23,998::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:24,034::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0220147 s, 46.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:25,464::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:25,466::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72182::DEBUG::2014-09-25 18:20:25,468::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:26,054::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:26,082::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0193472 s, 52.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:27,051::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:27,051::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:27,052::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:27,052::__init__::232::IOProcess::(_processLogs) (111146) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:27,061::__init__::232::IOProcess::(_processLogs) (111146) Queuing response >Thread-79::DEBUG::2014-09-25 18:20:27,063::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:20:27,088::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00687473 s, 81.7 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:27,090::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:27,090::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:27,090::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:27,090::__init__::232::IOProcess::(_processLogs) (111147) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:27,092::__init__::232::IOProcess::(_processLogs) (111147) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:27,094::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:27,094::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:27,094::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:27,095::__init__::232::IOProcess::(_processLogs) (111148) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:27,095::__init__::232::IOProcess::(_processLogs) (111148) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:27,096::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:27,096::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:27,096::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:27,096::__init__::232::IOProcess::(_processLogs) (111149) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:27,096::__init__::232::IOProcess::(_processLogs) (111149) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:27,097::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:27,098::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:27,098::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:27,098::__init__::232::IOProcess::(_processLogs) (111150) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:27,098::__init__::232::IOProcess::(_processLogs) (111150) Queuing response >Dummy-148::DEBUG::2014-09-25 18:20:28,102::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:28,132::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0184106 s, 55.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:28,484::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:28,486::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72183::DEBUG::2014-09-25 18:20:28,488::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:20:30,101::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:30,101::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:30,101::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:30,101::__init__::232::IOProcess::(_processLogs) (111151) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:30,106::__init__::232::IOProcess::(_processLogs) (111151) Queuing response >Thread-81::DEBUG::2014-09-25 18:20:30,108::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:20:30,140::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.0024421 s, 152 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:30,141::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:30,141::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:30,141::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:30,142::__init__::232::IOProcess::(_processLogs) (111152) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:30,147::__init__::232::IOProcess::(_processLogs) (111152) Queuing response >Dummy-148::DEBUG::2014-09-25 18:20:30,151::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:30,184::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0206364 s, 49.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:31,504::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:31,507::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72184::DEBUG::2014-09-25 18:20:31,509::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:32,204::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:32,227::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134124 s, 76.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:33,151::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72185::DEBUG::2014-09-25 18:20:33,152::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:20:33,152::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72185::DEBUG::2014-09-25 18:20:33,155::task::595::Storage.TaskManager.Task::(_updateState) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::moving from state init -> state preparing >Thread-72185::INFO::2014-09-25 18:20:33,155::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72185::ERROR::2014-09-25 18:20:33,155::task::866::Storage.TaskManager.Task::(_setError) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72185::DEBUG::2014-09-25 18:20:33,156::task::885::Storage.TaskManager.Task::(_run) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::Task._run: 9cdddc3c-8311-440c-b3be-f9e4e5359f59 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72185::DEBUG::2014-09-25 18:20:33,156::task::1217::Storage.TaskManager.Task::(stop) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::stopping in state preparing (force False) >Thread-72185::DEBUG::2014-09-25 18:20:33,156::task::993::Storage.TaskManager.Task::(_decref) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::ref 1 aborting True >Thread-72185::INFO::2014-09-25 18:20:33,157::task::1171::Storage.TaskManager.Task::(prepare) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::aborting: Task is aborted: u'' - code 100 >Thread-72185::DEBUG::2014-09-25 18:20:33,157::task::1176::Storage.TaskManager.Task::(prepare) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::Prepare: aborted: >Thread-72185::DEBUG::2014-09-25 18:20:33,157::task::993::Storage.TaskManager.Task::(_decref) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::ref 0 aborting True >Thread-72185::DEBUG::2014-09-25 18:20:33,157::task::928::Storage.TaskManager.Task::(_doAbort) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::Task._doAbort: force False >Thread-72185::DEBUG::2014-09-25 18:20:33,158::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72185::DEBUG::2014-09-25 18:20:33,158::task::595::Storage.TaskManager.Task::(_updateState) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::moving from state preparing -> state aborting >Thread-72185::DEBUG::2014-09-25 18:20:33,158::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::_aborting: recover policy none >Thread-72185::DEBUG::2014-09-25 18:20:33,158::task::595::Storage.TaskManager.Task::(_updateState) Task=`9cdddc3c-8311-440c-b3be-f9e4e5359f59`::moving from state aborting -> state failed >Thread-72185::DEBUG::2014-09-25 18:20:33,158::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72185::DEBUG::2014-09-25 18:20:33,159::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72185::ERROR::2014-09-25 18:20:33,159::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72185::DEBUG::2014-09-25 18:20:33,159::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:34,246::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:34,273::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0143753 s, 71.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:34,522::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:34,524::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72186::DEBUG::2014-09-25 18:20:34,526::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:36,293::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:36,337::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0311019 s, 32.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:37,100::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:37,100::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:37,101::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:37,101::__init__::232::IOProcess::(_processLogs) (111153) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:37,106::__init__::232::IOProcess::(_processLogs) (111153) Queuing response >Thread-79::DEBUG::2014-09-25 18:20:37,108::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:20:37,133::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00464246 s, 121 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:37,134::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:37,135::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:37,135::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:37,135::__init__::232::IOProcess::(_processLogs) (111154) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:37,141::__init__::232::IOProcess::(_processLogs) (111154) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:37,142::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:37,143::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:37,143::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:37,144::__init__::232::IOProcess::(_processLogs) (111155) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:37,144::__init__::232::IOProcess::(_processLogs) (111155) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:37,145::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:37,145::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:37,146::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:37,146::__init__::232::IOProcess::(_processLogs) (111156) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:37,146::__init__::232::IOProcess::(_processLogs) (111156) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:37,147::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:37,147::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:37,148::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:37,148::__init__::232::IOProcess::(_processLogs) (111157) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:37,149::__init__::232::IOProcess::(_processLogs) (111157) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:37,538::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:37,539::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72187::DEBUG::2014-09-25 18:20:37,541::task::595::Storage.TaskManager.Task::(_updateState) Task=`74a208d2-73e5-4b0e-96a9-7d337c2896db`::moving from state init -> state preparing >Thread-72187::INFO::2014-09-25 18:20:37,542::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72187::INFO::2014-09-25 18:20:37,542::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00464246', 'lastCheck': '0.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '39.4', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0024421', 'lastCheck': '7.4', 'valid': True}} >Thread-72187::DEBUG::2014-09-25 18:20:37,542::task::1191::Storage.TaskManager.Task::(prepare) Task=`74a208d2-73e5-4b0e-96a9-7d337c2896db`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00464246', 'lastCheck': '0.4', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '39.4', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.0024421', 'lastCheck': '7.4', 'valid': True}} >Thread-72187::DEBUG::2014-09-25 18:20:37,543::task::595::Storage.TaskManager.Task::(_updateState) Task=`74a208d2-73e5-4b0e-96a9-7d337c2896db`::moving from state preparing -> state finished >Thread-72187::DEBUG::2014-09-25 18:20:37,543::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72187::DEBUG::2014-09-25 18:20:37,543::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72187::DEBUG::2014-09-25 18:20:37,543::task::993::Storage.TaskManager.Task::(_decref) Task=`74a208d2-73e5-4b0e-96a9-7d337c2896db`::ref 0 aborting False >Thread-72187::DEBUG::2014-09-25 18:20:37,546::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:37,561::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:37,563::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72188::DEBUG::2014-09-25 18:20:37,564::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:38,356::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:38,383::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0159334 s, 64.3 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:40,150::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:40,150::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:40,151::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:40,151::__init__::232::IOProcess::(_processLogs) (111158) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:40,153::__init__::232::IOProcess::(_processLogs) (111158) Queuing response >Thread-81::DEBUG::2014-09-25 18:20:40,153::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:20:40,162::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.000338845 s, 1.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:40,163::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:40,164::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:40,164::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:40,164::__init__::232::IOProcess::(_processLogs) (111159) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:40,164::__init__::232::IOProcess::(_processLogs) (111159) Queuing response >Dummy-148::DEBUG::2014-09-25 18:20:40,403::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:40,425::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0136559 s, 75.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:40,611::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:40,613::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72189::DEBUG::2014-09-25 18:20:40,615::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:42,445::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:42,485::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0241851 s, 42.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:43,196::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72190::DEBUG::2014-09-25 18:20:43,197::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:20:43,197::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72190::DEBUG::2014-09-25 18:20:43,200::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::moving from state init -> state preparing >Thread-72190::INFO::2014-09-25 18:20:43,200::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72190::ERROR::2014-09-25 18:20:43,200::task::866::Storage.TaskManager.Task::(_setError) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72190::DEBUG::2014-09-25 18:20:43,201::task::885::Storage.TaskManager.Task::(_run) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::Task._run: fcb20d67-a510-4d98-a910-20a9f0097d60 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72190::DEBUG::2014-09-25 18:20:43,201::task::1217::Storage.TaskManager.Task::(stop) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::stopping in state preparing (force False) >Thread-72190::DEBUG::2014-09-25 18:20:43,201::task::993::Storage.TaskManager.Task::(_decref) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::ref 1 aborting True >Thread-72190::INFO::2014-09-25 18:20:43,202::task::1171::Storage.TaskManager.Task::(prepare) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::aborting: Task is aborted: u'' - code 100 >Thread-72190::DEBUG::2014-09-25 18:20:43,202::task::1176::Storage.TaskManager.Task::(prepare) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::Prepare: aborted: >Thread-72190::DEBUG::2014-09-25 18:20:43,202::task::993::Storage.TaskManager.Task::(_decref) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::ref 0 aborting True >Thread-72190::DEBUG::2014-09-25 18:20:43,202::task::928::Storage.TaskManager.Task::(_doAbort) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::Task._doAbort: force False >Thread-72190::DEBUG::2014-09-25 18:20:43,202::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72190::DEBUG::2014-09-25 18:20:43,203::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::moving from state preparing -> state aborting >Thread-72190::DEBUG::2014-09-25 18:20:43,203::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::_aborting: recover policy none >Thread-72190::DEBUG::2014-09-25 18:20:43,203::task::595::Storage.TaskManager.Task::(_updateState) Task=`fcb20d67-a510-4d98-a910-20a9f0097d60`::moving from state aborting -> state failed >Thread-72190::DEBUG::2014-09-25 18:20:43,203::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72190::DEBUG::2014-09-25 18:20:43,204::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72190::ERROR::2014-09-25 18:20:43,204::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72190::DEBUG::2014-09-25 18:20:43,204::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:43,627::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:43,629::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72191::DEBUG::2014-09-25 18:20:43,630::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:44,504::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:44,551::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0215456 s, 47.5 MB/s\n'; <rc> = 0 >Dummy-148::DEBUG::2014-09-25 18:20:46,576::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:46,615::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0252038 s, 40.6 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:46,644::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:46,648::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72192::DEBUG::2014-09-25 18:20:46,658::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:20:47,149::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:47,149::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:47,149::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:47,150::__init__::232::IOProcess::(_processLogs) (111160) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:47,153::__init__::232::IOProcess::(_processLogs) (111160) Queuing response >Thread-79::DEBUG::2014-09-25 18:20:47,153::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:20:47,172::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00451081 s, 125 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:47,173::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:47,173::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:47,174::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:47,174::__init__::232::IOProcess::(_processLogs) (111161) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:47,177::__init__::232::IOProcess::(_processLogs) (111161) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:47,179::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:47,180::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:47,180::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:47,180::__init__::232::IOProcess::(_processLogs) (111162) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:47,181::__init__::232::IOProcess::(_processLogs) (111162) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:47,181::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:47,182::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:47,182::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:47,183::__init__::232::IOProcess::(_processLogs) (111163) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:47,183::__init__::232::IOProcess::(_processLogs) (111163) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:47,183::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:47,184::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:47,185::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:47,185::__init__::232::IOProcess::(_processLogs) (111164) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:47,185::__init__::232::IOProcess::(_processLogs) (111164) Queuing response >Dummy-148::DEBUG::2014-09-25 18:20:48,805::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:48,842::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201301 s, 50.9 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:49,675::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:49,677::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72193::DEBUG::2014-09-25 18:20:49,679::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:20:50,166::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:50,166::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:50,167::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:50,167::__init__::232::IOProcess::(_processLogs) (111165) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:50,169::__init__::232::IOProcess::(_processLogs) (111165) Queuing response >Thread-81::DEBUG::2014-09-25 18:20:50,170::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:20:50,184::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00177511 s, 209 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:50,185::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:50,186::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:50,186::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:50,186::__init__::232::IOProcess::(_processLogs) (111166) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:50,192::__init__::232::IOProcess::(_processLogs) (111166) Queuing response >Dummy-148::DEBUG::2014-09-25 18:20:50,861::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:50,886::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0147945 s, 69.2 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:52,690::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:52,691::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72194::DEBUG::2014-09-25 18:20:52,694::task::595::Storage.TaskManager.Task::(_updateState) Task=`a28d84bd-d906-4e74-b816-7db72ca44a16`::moving from state init -> state preparing >Thread-72194::INFO::2014-09-25 18:20:52,694::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72194::INFO::2014-09-25 18:20:52,695::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00451081', 'lastCheck': '5.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '54.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00177511', 'lastCheck': '2.5', 'valid': True}} >Thread-72194::DEBUG::2014-09-25 18:20:52,695::task::1191::Storage.TaskManager.Task::(prepare) Task=`a28d84bd-d906-4e74-b816-7db72ca44a16`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00451081', 'lastCheck': '5.5', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '54.5', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00177511', 'lastCheck': '2.5', 'valid': True}} >Thread-72194::DEBUG::2014-09-25 18:20:52,695::task::595::Storage.TaskManager.Task::(_updateState) Task=`a28d84bd-d906-4e74-b816-7db72ca44a16`::moving from state preparing -> state finished >Thread-72194::DEBUG::2014-09-25 18:20:52,696::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72194::DEBUG::2014-09-25 18:20:52,696::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72194::DEBUG::2014-09-25 18:20:52,696::task::993::Storage.TaskManager.Task::(_decref) Task=`a28d84bd-d906-4e74-b816-7db72ca44a16`::ref 0 aborting False >Thread-72194::DEBUG::2014-09-25 18:20:52,698::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:52,714::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:52,715::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72195::DEBUG::2014-09-25 18:20:52,717::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:52,906::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:52,936::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0172198 s, 59.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:53,228::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:53,230::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72196::DEBUG::2014-09-25 18:20:53,230::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72196::DEBUG::2014-09-25 18:20:53,232::task::595::Storage.TaskManager.Task::(_updateState) Task=`27192988-5e11-4243-a55e-45dd72525f89`::moving from state init -> state preparing >Thread-72196::INFO::2014-09-25 18:20:53,232::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72196::ERROR::2014-09-25 18:20:53,233::task::866::Storage.TaskManager.Task::(_setError) Task=`27192988-5e11-4243-a55e-45dd72525f89`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72196::DEBUG::2014-09-25 18:20:53,233::task::885::Storage.TaskManager.Task::(_run) Task=`27192988-5e11-4243-a55e-45dd72525f89`::Task._run: 27192988-5e11-4243-a55e-45dd72525f89 (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72196::DEBUG::2014-09-25 18:20:53,234::task::1217::Storage.TaskManager.Task::(stop) Task=`27192988-5e11-4243-a55e-45dd72525f89`::stopping in state preparing (force False) >Thread-72196::DEBUG::2014-09-25 18:20:53,234::task::993::Storage.TaskManager.Task::(_decref) Task=`27192988-5e11-4243-a55e-45dd72525f89`::ref 1 aborting True >Thread-72196::INFO::2014-09-25 18:20:53,234::task::1171::Storage.TaskManager.Task::(prepare) Task=`27192988-5e11-4243-a55e-45dd72525f89`::aborting: Task is aborted: u'' - code 100 >Thread-72196::DEBUG::2014-09-25 18:20:53,234::task::1176::Storage.TaskManager.Task::(prepare) Task=`27192988-5e11-4243-a55e-45dd72525f89`::Prepare: aborted: >Thread-72196::DEBUG::2014-09-25 18:20:53,235::task::993::Storage.TaskManager.Task::(_decref) Task=`27192988-5e11-4243-a55e-45dd72525f89`::ref 0 aborting True >Thread-72196::DEBUG::2014-09-25 18:20:53,235::task::928::Storage.TaskManager.Task::(_doAbort) Task=`27192988-5e11-4243-a55e-45dd72525f89`::Task._doAbort: force False >Thread-72196::DEBUG::2014-09-25 18:20:53,235::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72196::DEBUG::2014-09-25 18:20:53,235::task::595::Storage.TaskManager.Task::(_updateState) Task=`27192988-5e11-4243-a55e-45dd72525f89`::moving from state preparing -> state aborting >Thread-72196::DEBUG::2014-09-25 18:20:53,235::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`27192988-5e11-4243-a55e-45dd72525f89`::_aborting: recover policy none >Thread-72196::DEBUG::2014-09-25 18:20:53,236::task::595::Storage.TaskManager.Task::(_updateState) Task=`27192988-5e11-4243-a55e-45dd72525f89`::moving from state aborting -> state failed >Thread-72196::DEBUG::2014-09-25 18:20:53,236::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72196::DEBUG::2014-09-25 18:20:53,236::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72196::ERROR::2014-09-25 18:20:53,236::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72196::DEBUG::2014-09-25 18:20:53,237::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:54,956::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:54,987::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0164384 s, 62.3 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:55,766::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:55,768::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72197::DEBUG::2014-09-25 18:20:55,770::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:57,007::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:57,029::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0134551 s, 76.1 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:57,185::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:57,186::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:57,186::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:57,186::__init__::232::IOProcess::(_processLogs) (111167) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:57,187::__init__::232::IOProcess::(_processLogs) (111167) Queuing response >Thread-79::DEBUG::2014-09-25 18:20:57,188::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:20:57,197::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.000313073 s, 1.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:20:57,198::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:57,198::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:57,198::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:57,199::__init__::232::IOProcess::(_processLogs) (111168) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:20:57,199::__init__::232::IOProcess::(_processLogs) (111168) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:57,200::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:57,200::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:57,200::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:57,201::__init__::232::IOProcess::(_processLogs) (111169) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:57,201::__init__::232::IOProcess::(_processLogs) (111169) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:57,202::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:57,202::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:57,202::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:57,203::__init__::232::IOProcess::(_processLogs) (111170) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:57,203::__init__::232::IOProcess::(_processLogs) (111170) Queuing response >Thread-29::DEBUG::2014-09-25 18:20:57,204::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:20:57,204::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:20:57,205::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:20:57,205::__init__::232::IOProcess::(_processLogs) (111171) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:20:57,205::__init__::232::IOProcess::(_processLogs) (111171) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:20:58,782::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:20:58,783::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72198::DEBUG::2014-09-25 18:20:58,785::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:20:59,049::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:20:59,081::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0213376 s, 48.0 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:21:00,194::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:00,194::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:00,195::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:00,195::__init__::232::IOProcess::(_processLogs) (111172) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:21:00,195::__init__::232::IOProcess::(_processLogs) (111172) Queuing response >Thread-81::DEBUG::2014-09-25 18:21:00,196::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:21:00,212::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00443943 s, 83.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:21:00,213::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:00,214::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:00,214::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:00,214::__init__::232::IOProcess::(_processLogs) (111173) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:21:00,219::__init__::232::IOProcess::(_processLogs) (111173) Queuing response >Dummy-148::DEBUG::2014-09-25 18:21:01,101::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:21:01,127::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0152663 s, 67.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:21:01,799::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:21:01,800::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72199::DEBUG::2014-09-25 18:21:01,802::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:21:03,147::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:21:03,175::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0174123 s, 58.8 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:21:03,259::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >Thread-72200::DEBUG::2014-09-25 18:21:03,260::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >JsonRpcServer::DEBUG::2014-09-25 18:21:03,261::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72200::DEBUG::2014-09-25 18:21:03,263::task::595::Storage.TaskManager.Task::(_updateState) Task=`49c51476-8046-4e5d-b422-18586143c11e`::moving from state init -> state preparing >Thread-72200::INFO::2014-09-25 18:21:03,263::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72200::ERROR::2014-09-25 18:21:03,264::task::866::Storage.TaskManager.Task::(_setError) Task=`49c51476-8046-4e5d-b422-18586143c11e`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72200::DEBUG::2014-09-25 18:21:03,264::task::885::Storage.TaskManager.Task::(_run) Task=`49c51476-8046-4e5d-b422-18586143c11e`::Task._run: 49c51476-8046-4e5d-b422-18586143c11e (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72200::DEBUG::2014-09-25 18:21:03,265::task::1217::Storage.TaskManager.Task::(stop) Task=`49c51476-8046-4e5d-b422-18586143c11e`::stopping in state preparing (force False) >Thread-72200::DEBUG::2014-09-25 18:21:03,265::task::993::Storage.TaskManager.Task::(_decref) Task=`49c51476-8046-4e5d-b422-18586143c11e`::ref 1 aborting True >Thread-72200::INFO::2014-09-25 18:21:03,265::task::1171::Storage.TaskManager.Task::(prepare) Task=`49c51476-8046-4e5d-b422-18586143c11e`::aborting: Task is aborted: u'' - code 100 >Thread-72200::DEBUG::2014-09-25 18:21:03,265::task::1176::Storage.TaskManager.Task::(prepare) Task=`49c51476-8046-4e5d-b422-18586143c11e`::Prepare: aborted: >Thread-72200::DEBUG::2014-09-25 18:21:03,266::task::993::Storage.TaskManager.Task::(_decref) Task=`49c51476-8046-4e5d-b422-18586143c11e`::ref 0 aborting True >Thread-72200::DEBUG::2014-09-25 18:21:03,266::task::928::Storage.TaskManager.Task::(_doAbort) Task=`49c51476-8046-4e5d-b422-18586143c11e`::Task._doAbort: force False >Thread-72200::DEBUG::2014-09-25 18:21:03,266::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72200::DEBUG::2014-09-25 18:21:03,266::task::595::Storage.TaskManager.Task::(_updateState) Task=`49c51476-8046-4e5d-b422-18586143c11e`::moving from state preparing -> state aborting >Thread-72200::DEBUG::2014-09-25 18:21:03,266::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`49c51476-8046-4e5d-b422-18586143c11e`::_aborting: recover policy none >Thread-72200::DEBUG::2014-09-25 18:21:03,267::task::595::Storage.TaskManager.Task::(_updateState) Task=`49c51476-8046-4e5d-b422-18586143c11e`::moving from state aborting -> state failed >Thread-72200::DEBUG::2014-09-25 18:21:03,267::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72200::DEBUG::2014-09-25 18:21:03,267::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72200::ERROR::2014-09-25 18:21:03,267::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72200::DEBUG::2014-09-25 18:21:03,268::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:21:04,815::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:21:04,816::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72201::DEBUG::2014-09-25 18:21:04,818::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-72168::DEBUG::2014-09-25 18:21:05,163::hsm::2441::Storage.HSM::(connectStorageServer) prefetch failed: {u'3573f707-d0c4-4dc9-b09f-dca9a3506a4f': <function findDomain at 0x14be758>, u'fc587b6a-bac3-4668-99de-7f6fd6acce7f': <function findDomain at 0x14be758>, u'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': <function findDomain at 0x14be758>} >Traceback (most recent call last): > File "/usr/share/vdsm/storage/hsm.py", line 2438, in connectStorageServer > doms = self.__prefetchDomains(domType, conObj) > File "/usr/share/vdsm/storage/hsm.py", line 2375, in __prefetchDomains > goop.glob.glob(os.path.join(lPath, uuidPatern))) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-72168::DEBUG::2014-09-25 18:21:05,163::hsm::2452::Storage.HSM::(connectStorageServer) knownSDs: {3573f707-d0c4-4dc9-b09f-dca9a3506a4f: storage.nfsSD.findDomain, fc587b6a-bac3-4668-99de-7f6fd6acce7f: storage.nfsSD.findDomain, eb633636-3019-47a9-9f5a-fc81f8dfd7dc: storage.nfsSD.findDomain} >Thread-72168::INFO::2014-09-25 18:21:05,164::logUtils::47::dispatcher::(wrapper) Run and protect: connectStorageServer, Return response: {'statuslist': [{'status': 0, 'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8'}]} >Thread-72168::DEBUG::2014-09-25 18:21:05,164::task::1191::Storage.TaskManager.Task::(prepare) Task=`391a2aef-b5dc-420b-b2c9-2a709f5bf90c`::finished: {'statuslist': [{'status': 0, 'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8'}]} >Thread-72168::DEBUG::2014-09-25 18:21:05,164::task::595::Storage.TaskManager.Task::(_updateState) Task=`391a2aef-b5dc-420b-b2c9-2a709f5bf90c`::moving from state preparing -> state finished >Thread-72168::DEBUG::2014-09-25 18:21:05,164::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72168::DEBUG::2014-09-25 18:21:05,164::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72168::DEBUG::2014-09-25 18:21:05,165::task::993::Storage.TaskManager.Task::(_decref) Task=`391a2aef-b5dc-420b-b2c9-2a709f5bf90c`::ref 0 aborting False >Thread-72168::DEBUG::2014-09-25 18:21:05,165::__init__::498::jsonrpc.JsonRpcServer::(_serveRequest) Return 'StoragePool.connectStorageServer' in bridge with [{'status': 0, 'id': u'83fd005b-3471-44cb-99d9-bc14b3dc1ae8'}] >Thread-72168::DEBUG::2014-09-25 18:21:05,165::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:21:05,194::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:21:05,463::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.259564 s, 3.9 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:21:07,207::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:07,207::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:07,207::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:07,208::__init__::232::IOProcess::(_processLogs) (111174) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:21:07,211::__init__::232::IOProcess::(_processLogs) (111174) Queuing response >Thread-79::DEBUG::2014-09-25 18:21:07,213::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:21:07,229::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00189863 s, 296 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:21:07,230::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:07,230::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:07,230::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:07,231::__init__::232::IOProcess::(_processLogs) (111175) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:21:07,235::__init__::232::IOProcess::(_processLogs) (111175) Queuing response >Thread-29::DEBUG::2014-09-25 18:21:07,236::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:07,237::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:07,237::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:07,238::__init__::232::IOProcess::(_processLogs) (111176) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:21:07,238::__init__::232::IOProcess::(_processLogs) (111176) Queuing response >Thread-29::DEBUG::2014-09-25 18:21:07,239::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:07,239::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:07,240::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:07,240::__init__::232::IOProcess::(_processLogs) (111177) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:21:07,240::__init__::232::IOProcess::(_processLogs) (111177) Queuing response >Thread-29::DEBUG::2014-09-25 18:21:07,241::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:07,241::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:07,242::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:07,242::__init__::232::IOProcess::(_processLogs) (111178) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:21:07,243::__init__::232::IOProcess::(_processLogs) (111178) Queuing response >Dummy-148::DEBUG::2014-09-25 18:21:07,482::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:21:07,521::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0269679 s, 38.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:21:07,829::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:21:07,831::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72202::DEBUG::2014-09-25 18:21:07,834::task::595::Storage.TaskManager.Task::(_updateState) Task=`cd9bbe09-da87-4f7a-8bf1-1451b77a21c8`::moving from state init -> state preparing >Thread-72202::INFO::2014-09-25 18:21:07,834::logUtils::44::dispatcher::(wrapper) Run and protect: repoStats(options=None) >Thread-72202::INFO::2014-09-25 18:21:07,834::logUtils::47::dispatcher::(wrapper) Run and protect: repoStats, Return response: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00189863', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '69.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00443943', 'lastCheck': '7.6', 'valid': True}} >Thread-72202::DEBUG::2014-09-25 18:21:07,835::task::1191::Storage.TaskManager.Task::(prepare) Task=`cd9bbe09-da87-4f7a-8bf1-1451b77a21c8`::finished: {'eb633636-3019-47a9-9f5a-fc81f8dfd7dc': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00189863', 'lastCheck': '0.6', 'valid': True}, 'fc587b6a-bac3-4668-99de-7f6fd6acce7f': {'code': 358, 'version': -1, 'acquired': False, 'delay': '0', 'lastCheck': '69.7', 'valid': False}, '3573f707-d0c4-4dc9-b09f-dca9a3506a4f': {'code': 0, 'version': 0, 'acquired': True, 'delay': '0.00443943', 'lastCheck': '7.6', 'valid': True}} >Thread-72202::DEBUG::2014-09-25 18:21:07,835::task::595::Storage.TaskManager.Task::(_updateState) Task=`cd9bbe09-da87-4f7a-8bf1-1451b77a21c8`::moving from state preparing -> state finished >Thread-72202::DEBUG::2014-09-25 18:21:07,835::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72202::DEBUG::2014-09-25 18:21:07,835::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72202::DEBUG::2014-09-25 18:21:07,836::task::993::Storage.TaskManager.Task::(_decref) Task=`cd9bbe09-da87-4f7a-8bf1-1451b77a21c8`::ref 0 aborting False >Thread-72202::DEBUG::2014-09-25 18:21:07,838::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:21:07,853::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:21:07,854::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72203::DEBUG::2014-09-25 18:21:07,856::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-72172::WARNING::2014-09-25 18:21:08,180::fileSD::688::Storage.scanDomains::(collectMetaFiles) Could not collect metadata file for domain path /rhev/data-center/mnt/pb-rh35.rhev.lab.eng.brq.redhat.com:_var_lib_exports_iso >Traceback (most recent call last): > File "/usr/share/vdsm/storage/fileSD.py", line 674, in collectMetaFiles > sd.DOMAIN_META_DATA)) > File "/usr/share/vdsm/storage/outOfProcess.py", line 89, in glob > return self._iop.glob(pattern) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 460, in glob > return self._sendCommand("glob", {"pattern": pattern}, self.timeout) > File "/usr/lib/python2.7/site-packages/ioprocess/__init__.py", line 345, in _sendCommand > raise Timeout(os.strerror(errno.ETIMEDOUT)) >Timeout: Connection timed out >Thread-80::ERROR::2014-09-25 18:21:08,181::sdc::143::Storage.StorageDomainCache::(_findDomain) domain fc587b6a-bac3-4668-99de-7f6fd6acce7f not found >Traceback (most recent call last): > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Thread-80::ERROR::2014-09-25 18:21:08,181::domainMonitor::256::Storage.DomainMonitorThread::(_monitorDomain) Error while collecting domain fc587b6a-bac3-4668-99de-7f6fd6acce7f monitoring information >Traceback (most recent call last): > File "/usr/share/vdsm/storage/domainMonitor.py", line 232, in _monitorDomain > self.domain.selftest() > File "/usr/share/vdsm/storage/sdc.py", line 49, in __getattr__ > return getattr(self.getRealDomain(), attrName) > File "/usr/share/vdsm/storage/sdc.py", line 52, in getRealDomain > return self._cache._realProduce(self._sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 122, in _realProduce > domain = self._findDomain(sdUUID) > File "/usr/share/vdsm/storage/sdc.py", line 141, in _findDomain > dom = findMethod(sdUUID) > File "/usr/share/vdsm/storage/nfsSD.py", line 122, in findDomain > return NfsStorageDomain(NfsStorageDomain.findDomainPath(sdUUID)) > File "/usr/share/vdsm/storage/nfsSD.py", line 112, in findDomainPath > raise se.StorageDomainDoesNotExist(sdUUID) >StorageDomainDoesNotExist: Storage domain does not exist: ('fc587b6a-bac3-4668-99de-7f6fd6acce7f',) >Dummy-148::DEBUG::2014-09-25 18:21:09,541::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:21:09,572::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0201508 s, 50.8 MB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:21:10,221::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:10,221::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:10,221::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:10,222::__init__::232::IOProcess::(_processLogs) (111179) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:21:10,230::__init__::232::IOProcess::(_processLogs) (111179) Queuing response >Thread-81::DEBUG::2014-09-25 18:21:10,231::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35export/3573f707-d0c4-4dc9-b09f-dca9a3506a4f/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-81::DEBUG::2014-09-25 18:21:10,256::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n371 bytes (371 B) copied, 0.00271019 s, 137 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:21:10,258::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:10,258::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:10,258::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:10,258::__init__::232::IOProcess::(_processLogs) (111180) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:21:10,260::__init__::232::IOProcess::(_processLogs) (111180) Queuing response >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:21:10,910::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:21:10,911::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72204::DEBUG::2014-09-25 18:21:10,913::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:21:11,592::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:21:11,620::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.017039 s, 60.1 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:21:13,292::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:21:13,293::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72205::DEBUG::2014-09-25 18:21:13,293::__init__::467::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'StoragePool.getSpmStatus' in bridge with {u'storagepoolID': u'00000002-0002-0002-0002-0000000001ad'} >Thread-72205::DEBUG::2014-09-25 18:21:13,295::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::moving from state init -> state preparing >Thread-72205::INFO::2014-09-25 18:21:13,296::logUtils::44::dispatcher::(wrapper) Run and protect: getSpmStatus(spUUID=u'00000002-0002-0002-0002-0000000001ad', options=None) >Thread-72205::ERROR::2014-09-25 18:21:13,296::task::866::Storage.TaskManager.Task::(_setError) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::Unexpected error >Traceback (most recent call last): > File "/usr/share/vdsm/storage/task.py", line 873, in _run > return fn(*args, **kargs) > File "/usr/share/vdsm/logUtils.py", line 45, in wrapper > res = f(*args, **kwargs) > File "/usr/share/vdsm/storage/hsm.py", line 611, in getSpmStatus > status = self._getSpmStatusInfo(pool) > File "/usr/share/vdsm/storage/hsm.py", line 605, in _getSpmStatusInfo > (pool.spmRole,) + pool.getSpmStatus())) > File "/usr/share/vdsm/storage/sp.py", line 126, in getSpmStatus > return self._backend.getSpmStatus() > File "/usr/share/vdsm/storage/spbackends.py", line 416, in getSpmStatus > lVer, spmId = self.masterDomain.inquireClusterLock() > File "/usr/share/vdsm/storage/sd.py", line 515, in inquireClusterLock > return self._clusterLock.inquire() > File "/usr/share/vdsm/storage/clusterlock.py", line 149, in inquire > raise InquireNotSupportedError() >InquireNotSupportedError >Thread-72205::DEBUG::2014-09-25 18:21:13,296::task::885::Storage.TaskManager.Task::(_run) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::Task._run: 1c402b45-0d91-4a50-82d3-86654a0aa09d (u'00000002-0002-0002-0002-0000000001ad',) {} failed - stopping task >Thread-72205::DEBUG::2014-09-25 18:21:13,297::task::1217::Storage.TaskManager.Task::(stop) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::stopping in state preparing (force False) >Thread-72205::DEBUG::2014-09-25 18:21:13,297::task::993::Storage.TaskManager.Task::(_decref) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::ref 1 aborting True >Thread-72205::INFO::2014-09-25 18:21:13,297::task::1171::Storage.TaskManager.Task::(prepare) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::aborting: Task is aborted: u'' - code 100 >Thread-72205::DEBUG::2014-09-25 18:21:13,297::task::1176::Storage.TaskManager.Task::(prepare) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::Prepare: aborted: >Thread-72205::DEBUG::2014-09-25 18:21:13,298::task::993::Storage.TaskManager.Task::(_decref) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::ref 0 aborting True >Thread-72205::DEBUG::2014-09-25 18:21:13,298::task::928::Storage.TaskManager.Task::(_doAbort) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::Task._doAbort: force False >Thread-72205::DEBUG::2014-09-25 18:21:13,298::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72205::DEBUG::2014-09-25 18:21:13,298::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::moving from state preparing -> state aborting >Thread-72205::DEBUG::2014-09-25 18:21:13,299::task::550::Storage.TaskManager.Task::(__state_aborting) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::_aborting: recover policy none >Thread-72205::DEBUG::2014-09-25 18:21:13,299::task::595::Storage.TaskManager.Task::(_updateState) Task=`1c402b45-0d91-4a50-82d3-86654a0aa09d`::moving from state aborting -> state failed >Thread-72205::DEBUG::2014-09-25 18:21:13,299::resourceManager::940::Storage.ResourceManager.Owner::(releaseAll) Owner.releaseAll requests {} resources {} >Thread-72205::DEBUG::2014-09-25 18:21:13,299::resourceManager::977::Storage.ResourceManager.Owner::(cancelAll) Owner.cancelAll requests {} >Thread-72205::ERROR::2014-09-25 18:21:13,299::dispatcher::79::Storage.Dispatcher::(wrapper) >Traceback (most recent call last): > File "/usr/share/vdsm/storage/dispatcher.py", line 71, in wrapper > result = ctask.prepare(func, *args, **kwargs) > File "/usr/share/vdsm/storage/task.py", line 103, in wrapper > return m(self, *a, **kw) > File "/usr/share/vdsm/storage/task.py", line 1179, in prepare > raise self.error >InquireNotSupportedError >Thread-72205::DEBUG::2014-09-25 18:21:13,300::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:21:13,640::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:21:13,662::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0132929 s, 77.0 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:21:13,926::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:21:13,927::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72206::DEBUG::2014-09-25 18:21:13,929::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Dummy-148::DEBUG::2014-09-25 18:21:15,682::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:21:15,712::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0181187 s, 56.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:21:16,942::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:21:16,946::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >Thread-72207::DEBUG::2014-09-25 18:21:16,951::stompReactor::163::yajsonrpc.StompServer::(send) Sending response >Thread-29::DEBUG::2014-09-25 18:21:17,243::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:17,243::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:17,243::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:17,244::__init__::232::IOProcess::(_processLogs) (111181) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:21:17,257::__init__::232::IOProcess::(_processLogs) (111181) Queuing response >Thread-79::DEBUG::2014-09-25 18:21:17,258::fileSD::261::Storage.Misc.excCmd::(getReadDelay) /usr/bin/dd if=/rhev/data-center/mnt/10.34.63.202:_mnt_export_nfs_lv3_pbenas_35/eb633636-3019-47a9-9f5a-fc81f8dfd7dc/dom_md/metadata iflag=direct of=/dev/null bs=4096 count=1 (cwd None) >Thread-79::DEBUG::2014-09-25 18:21:17,279::fileSD::261::Storage.Misc.excCmd::(getReadDelay) SUCCESS: <err> = '0+1 records in\n0+1 records out\n562 bytes (562 B) copied, 0.00753362 s, 74.6 kB/s\n'; <rc> = 0 >Thread-29::DEBUG::2014-09-25 18:21:17,280::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:17,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:17,280::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:17,281::__init__::232::IOProcess::(_processLogs) (111182) Got request for method 'statvfs' >Thread-29::DEBUG::2014-09-25 18:21:17,285::__init__::232::IOProcess::(_processLogs) (111182) Queuing response >Thread-29::DEBUG::2014-09-25 18:21:17,286::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:17,287::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:17,287::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:17,288::__init__::232::IOProcess::(_processLogs) (111183) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:21:17,288::__init__::232::IOProcess::(_processLogs) (111183) Queuing response >Thread-29::DEBUG::2014-09-25 18:21:17,288::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:17,289::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:17,290::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:17,290::__init__::232::IOProcess::(_processLogs) (111184) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:21:17,290::__init__::232::IOProcess::(_processLogs) (111184) Queuing response >Thread-29::DEBUG::2014-09-25 18:21:17,291::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:17,291::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:17,292::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:17,292::__init__::232::IOProcess::(_processLogs) (111185) Got request for method 'access' >Thread-29::DEBUG::2014-09-25 18:21:17,293::__init__::232::IOProcess::(_processLogs) (111185) Queuing response >Dummy-148::DEBUG::2014-09-25 18:21:17,733::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:21:17,755::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.013468 s, 76.0 MB/s\n'; <rc> = 0 >Thread-80::DEBUG::2014-09-25 18:21:18,182::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.sdc.refreshStorage) >Thread-80::DEBUG::2014-09-25 18:21:18,182::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-80::DEBUG::2014-09-25 18:21:18,183::misc::741::Storage.SamplingMethod::(__call__) Trying to enter sampling method (storage.iscsi.rescan) >Thread-80::DEBUG::2014-09-25 18:21:18,183::misc::743::Storage.SamplingMethod::(__call__) Got in to sampling method >Thread-80::DEBUG::2014-09-25 18:21:18,183::iscsi::403::Storage.ISCSI::(rescan) Performing SCSI scan, this will take up to 30 seconds >Thread-80::DEBUG::2014-09-25 18:21:18,184::iscsiadm::92::Storage.Misc.excCmd::(_runCmd) /usr/bin/sudo -n /sbin/iscsiadm -m session -R (cwd None) >Thread-80::DEBUG::2014-09-25 18:21:18,204::misc::751::Storage.SamplingMethod::(__call__) Returning last result >Thread-80::DEBUG::2014-09-25 18:21:18,205::multipath::110::Storage.Misc.excCmd::(rescan) /usr/bin/sudo -n /sbin/multipath (cwd None) >Thread-80::DEBUG::2014-09-25 18:21:18,273::multipath::110::Storage.Misc.excCmd::(rescan) SUCCESS: <err> = ''; <rc> = 0 >Thread-80::DEBUG::2014-09-25 18:21:18,274::lvm::489::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-80::DEBUG::2014-09-25 18:21:18,274::lvm::491::Storage.OperationMutex::(_invalidateAllPvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-80::DEBUG::2014-09-25 18:21:18,274::lvm::500::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' got the operation mutex >Thread-80::DEBUG::2014-09-25 18:21:18,275::lvm::502::Storage.OperationMutex::(_invalidateAllVgs) Operation 'lvm invalidate operation' released the operation mutex >Thread-80::DEBUG::2014-09-25 18:21:18,275::lvm::520::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' got the operation mutex >Thread-80::DEBUG::2014-09-25 18:21:18,275::lvm::522::Storage.OperationMutex::(_invalidateAllLvs) Operation 'lvm invalidate operation' released the operation mutex >Thread-80::DEBUG::2014-09-25 18:21:18,276::misc::751::Storage.SamplingMethod::(__call__) Returning last result >Thread-29::DEBUG::2014-09-25 18:21:18,280::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:18,280::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:18,280::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:18,280::__init__::232::IOProcess::(_processLogs) (111186) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:21:18,281::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:18,281::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:18,281::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:18,281::__init__::232::IOProcess::(_processLogs) (111187) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:21:18,282::__init__::232::IOProcess::(_processLogs) Receiving request... >Thread-29::DEBUG::2014-09-25 18:21:18,282::__init__::232::IOProcess::(_processLogs) Queuing request in the thread pool... >Thread-29::DEBUG::2014-09-25 18:21:18,283::__init__::232::IOProcess::(_processLogs) (111186) Queuing response >Thread-29::DEBUG::2014-09-25 18:21:18,283::__init__::232::IOProcess::(_processLogs) Extracting request information... >Thread-29::DEBUG::2014-09-25 18:21:18,284::__init__::232::IOProcess::(_processLogs) (111188) Got request for method 'glob' >Thread-29::DEBUG::2014-09-25 18:21:18,285::__init__::232::IOProcess::(_processLogs) (111188) Queuing response >Dummy-148::DEBUG::2014-09-25 18:21:19,775::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) dd if=/rhev/data-center/00000002-0002-0002-0002-0000000001ad/mastersd/dom_md/inbox iflag=direct,fullblock count=1 bs=1024000 (cwd None) >Dummy-148::DEBUG::2014-09-25 18:21:19,804::storage_mailbox::731::Storage.Misc.excCmd::(_checkForMail) SUCCESS: <err> = '1+0 records in\n1+0 records out\n1024000 bytes (1.0 MB) copied, 0.0191474 s, 53.5 MB/s\n'; <rc> = 0 >JsonRpc (StompReactor)::DEBUG::2014-09-25 18:21:19,966::stompReactor::98::Broker.StompAdapter::(handle_frame) Handling message <StompFrame command='SEND'> >JsonRpcServer::DEBUG::2014-09-25 18:21:19,968::__init__::504::jsonrpc.JsonRpcServer::(serve_requests) Waiting for request >